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ABSTRACT 
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Chapter I. Introduction and Overview 



Everyone who works in education is well aware that we are in an era of accountability. 
Like never before, schools and teachers must demonstrate how their instruction and services 
have affected student outcomes. In many cases, the very existence and funding of an education 
program depends on its ability to demonstrate successful outcomes. To the traditional tools of 
teaching, such as the blackboard, chalk and books, we now must add the computer and the 
database. 

In adult education, accountability is defined through the National Reporting System 
(NRS), an outcome-based reporting system established to meet the requirements of Title II of the 
Workforce Investment Act (WIA), which authorizes a state grant program for the delivery of 
adult basic education (ABE), adult secondary education (ASE) and English as a second language 
(ESL) instruction. The NRS defines outcome, participation and student descriptive measures, 
specifies the methodologies through which local providers are to collect these measures and 
establishes state reporting requirements. Exhibit 1-1 summarizes the core (required) and 
optional NRS outcome measures and definitions. 



Exhibit 1-1 

Summary of NRS Measures and Definitions 



Topic Measures Categories or Definitions 

Core Outcome Measures 


Educational Gains 




❖ Educational gains ❖ Educational functioning levels in 

reading, writing, speaking and 
listening and functional areas. 


Followup Measures 




❖ Entered employment ❖ Learners who obtain a job by the end 

of the first quarter after exit quarter 

♦> Retained employment ♦> Learners who remain employed in the 

third quarter after program exit 

❖ Receipt of secondary school diploma ♦> Learners who obtain a GED, 

or GED secondary school diploma or 

recognized equivalent 

❖ Placement in postsecondary ♦> Learners enrolling in a postsecondary 

education or training educational or occupational skills 

program building on prior services or 
training received 


j§» f|f '*! 


m 

1 


Core Descriptive and Participation Measures 


Demographics 


. Wy-: ••• 


❖ Ethnicity ❖ American Indian or Alaskan Native, 

Asian, Native Hawaiian or Pacific 
Islander, Black or African American 
(non-Hispanic), Hispanic or Latino, 
White (non-Hispanic) 

❖ Gender ♦> Male, female 

♦> Age ❖ Date of birth 


Status and Goals 




❖ Labor force status ❖ Employed, not employed, not in labor 

force 

❖ Public assistance status ❖ Receiving or not receiving assistance 

♦> Rural residency ❖ Rural, not rural 

❖ Disability status ♦> Disabled, not disabled 


"" Ify 


# . 
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Exhibit 1-1 (Continued) 

Summary of NRS Measures and Definitions 



Topic Measures Categories or Definitions 


Core Descriptive and Participation Measures (Continued) 


Student Participation 

i 'BK| 

v ■■■ 
• '»■: . 

■ HH 


1 ❖ Learner main and secondary ❖ Obtain a job, retain current job, 

| reasons or goals for attending improve current job, earn a 

j secondary school diploma or GED, 

; enter posLsecondary education or 

j training, improve basic literacy skills, 

| improve English language skills, 

j citizenship, work-based project 

learner goal, other personal goal 

❖ Contact hours ♦> Number of hours of instructional 

j activity 

❖ Program enrollment type ❖ ABE, ESL, ASE, family literacy, 

workplace program, homeless 
program, correctional facilities, 
community corrections programs, 
other institutional program 


Secondary Outcome and Student Status Measures (Optional) 


Employment 

-1 & 


❖ Reduction in receipt of public ❖ Students whose TANF or equivalent 

assistance public assistance grant is reduced or 

eliminated due to employment 


_ 

Work* based Project Learner 
Achievement ^ 

, , fl 


❖ Met work-based project learner goal ♦> Achieved skills for work-based project 

learner activity (activity of at least 12 
hours and no more than 30 hours of 
instruction to teach specific 
workplace skill) 


Community • j| 

i 

m 


❖ Achieved citizenship skills ♦> Achieve the skills needed to pass the 

citizenship exam 

❖ Voting behavior ♦> Learner registers to vote or votes for 

the first time 

❖ General involvement in community ❖ Learner increases involvement in 

activities community activities 


Famiiy 

fig 

► V 


❖ Involvement in children’s education ❖ Learner increases help given for 

children’s school work, contact with 
teachers to discuss education, and 
involvement in children’s school 

❖ Involvement in children’s literacy- ♦> Learner increases the amount read to 

related activities children, visits libraries, or purchases 

books or magazines for children 


Student Status 


❖ Low income status ❖ Low income, not low income 

❖ Displaced homemaker ❖ Displaced homemaker, not displaced 

homemaker 

❖ Single parent status ❖ Single parent, not single parent 

❖ Dislocated worker ❖ Dislocated worker, not dislocated 

worker 

❖ Learning disabled adult ♦> Learning disabled, not learning 

disabled 


- ; 1 | ; . 

IllilliS 

fa fa- ' 

- i i wiiii 



The Office of Vocational and Adult Education (OVAE) of the U.S. Department of 
Education developed the NRS through a collaborative process with state directors of adult 
education, other literacy providers, and WIA-defined partner agencies. Through a series of 
projects, OVAE established overall requirements and guidelines for the system ( Measures and 
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Methods for the National Reporting System for Adult Education: Implementation Guidelines, 
March 2001); set the procedures for the follow-up survey ( Guidelines for Conducting the 
Follow-up Survey, March 2001); sponsored a national training conference and developed state 
training materials to introduce the NRS ( National Reporting System Professional Development 
Materials, July 1999); and developed a web-based training site, NRS Online. 

The focus of this past work was on creating the NRS, informing adult education 
providers of its requirements and providing initial training on implemeitation. Now that the 
NRS is operational, states have some experience implementing procedures and collecting NRS 
measures. The focus of NRS activities now is to improve the quality of data collected by local 
programs and to promote the use of NRS data for program management and improvement at the 
state and local level. To this end, OVAE began a new project in the fall of 2001, Promoting the 
Quality and Use of NRS Data. 

Purpose of This Guide 

While all states and adult education programs are now implementing the NRS, the level 
of understanding of the system and the state and local procedures in place for collecting NRS 
data vary substantially. This variation and lack of standardization affect the quality of NRS data 
and limit its use as indicators of the adult education program and for program improvement. The 
purpose of this Guide for Improving NRS Data Quality is to provide practical, easy to understand 
information and materials for state and local staff that clarify the data collection process and 
explain NRS procedures for ensuring the collection of quality data. The guide is also designed to 
help local and state adult education program staff: 

1 . Identify the factors that contribute to quality data at each step of the data collection 

process; 

2. Distinguish between quality data and poor data; 

3. Identify problems and strengths in their data collection processes; 

4. Identify strategies to improve their data collection; and 

5. Understand how to use data for program improvement. 

The content of the guide assumes a basic knowledge of the NRS measures and 
procedures. It does not provide a comprehensive review or explanation of NRS policy and 
requirements, but focuses on the collection of the measures most critical to the NRS: the core 
outcome measures of educational gain-, the follow-up measures (entered employment, retained 
employment, receipt of secondary credential and placement in postsecondary education); intake 
and goal setting, since this process affects the follow-up measures; and the data collection 
process. It also reviews training and professional development needs for implementing the NRS 
and the state role in setting data policies. 
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Components of Data Quality 

The focus of this guide is to help programs and states improve their understanding of, and 
procedures for, collecting data in the NRS framework. Following these procedures will improve 
the overall quality of data within the NRS. Underlying the model of data quality used in this 
guide are the basic measurement concepts of reliability — following the same procedures 
consistently over time — and validity — ensuring the measures truly represent their underlying 
concepts. Other components of data quality the guide reflects include: 

• Objectivity. Quality data are information that is accurate and unbiased and presented in a 
clear, complete, well-documented manner. Objectivity is achieved by using appropriate 
data sources and sound analytical techniques, by using proven methods, and by carefully 
reviewing the content of all information and reports. 

• Integrity. Data with integrity is information that is not compromised through corruption 
or falsification. 

• Transparency. Transparent data involve having a clear description of methods, data 
sources, assumptions, outcomes, and related information that allow users to understand 
the data. 

• Reproducibility. Quality data can be reproduced by others by using the documented 
methods, assumptions, and data sources to achieve comparable findings. 

• Utility. Quality data are information that is useful and available to its intended audience. 

The requirements, recommendations and examples in this guide will help state and local 
programs improve their data quality within each of these dimensions. 

Users of This Guide 

Improving data quality requires making change at the source, where data are actually 
collected. Consequently, this guide is designed for adult education staff who are directly on the 
front lines of data collection — local program staff who instruct students, assist them in setting 
their goals, assess them and track their outcomes. Local program staff members not only collect 
the data that are the basis for the entire delivery system, but they also can benefit the most from 
using the data they collect. Improving data collection at the local level is the key to improving 
quality in the NRS and use of NRS data for program improvement. 

The state plays a major role in the NRS, as it is responsible for setting policies, providing 
training to local staff and monitoring data collection at the local level. State staff can use the 
guide to evaluate whether the state has procedures and policies that promote quality data and 
conform to NRS requirements. The guide also provides detailed guidance on how to review 
local program procedures to ensure collection of quality NRS data. 
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Staff development professionals and others who train local program staff on data 
collection will also find the guide useful. It provides step-by-step procedures on setting up a data 
collection system for collecting NRS measures that can be used as content in training on the 
NRS. The guide contains procedural self-evaluation forms that can be used as training tools to 
raise awareness of data collection issues. 

Overview of the Guide 

The next chapter of this guide, Quality Data: Why Good Data Matter , discusses the 
importance of collecting data, addressing the issue of why collecting quality data is important. 

Its main purpose is to demonstrate that collecting and reviewing data are valuable, essential 
activities of program management, not just “completing some meaningless forms for the state.” 
The chapter begins by defining data, explaining how programs and states can use data for 
program improvement and provides a model relating data quality and data use. The chapter also 
explains the importance of data quality at the federal and state levels. 

Chapter III, The Data Collection Process , analyzes the local program data collection 
process as a system. It offers a model data collection process, and reviews the key components 
of the model that are essential to producing quality data. The chapter also presents the main 
processes needed to put a quality data collection system in place and maintain its effective 
operation. 

Collecting NRS Outcome Measures , Chapter IV, reviews procedures for collecting each 
of the core NRS outcome measures: educational gain and the four follow-up measures. The 
section on follow-up measures explains both survey and data matching procedures. The chapter 
also reviews intake and goal setting procedures and discusses procedures for training local 
program staff to collect NRS measures. 

The final chapter, The State Role in Data Quality , reviews state responsibilities in the 
NRS that are essential to collecting quality data. Issues discussed in this chapter are state 
assessment policies, follow-up methods, training and data auditing. The chapter also offers 
suggestions on ways to provide local program oversight. 

To enhance the use of this guide as a technical assistance tool, Chapters III through V 
illustrate the concepts covered in each chapter in four additional ways: 

• Indicators of performance. This section offers methods for reviewing data collection 
procedures for each measure and process. These methods will help gauge whether 
procedures are working well and identify problem areas. 

• Self-evaluation forms . The self-evaluations follow the discussion of each substantive 
topic and contain checklists of the policies and procedures to use to enhance data quality, 
worksheets for identifying areas of strength and weakness and a planning chart to help 
plan improvements. 
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• Sample data tables. Following the discussion of most topics, we provide sample data 
tables, highlighting issues and potential problems discussed in the section. The tables 
illustrate how you can use data to pinpoint problems and understand whether data 
collection procedures are working as intended. 

• Case studies and vignettes. Each section offers a real example of how a local adult 
education program or a state addressed the issues discussed in the section to improve data 
quality. 

Case studies contain contact information to allow you to obtain further information on the issue. 
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Chapter II. Quality Data: Why Good Data Matter 

An accountability system, such as the NRS, relies on quality data for its integrity. 
Collecting and transmitting data is both an art and science that most literacy programs need help 
developing. The questions that public and private supporters have today about the literacy 
programs they support can only be answered with data. Program staff that hopes to improve its 
programs must have data to analyze for decision-making. No one wants to worry about the 
accuracy of the data when making programmatic or funding decisions, so it is crucial to have 
quality data. 

Before we get into the details on how to collect quality data, we begin by discussing why 
data quality is important. We define data, discuss why program staff sometimes guess wrong 
about the effect of their programs and point out how data can help improve a program. We 
present a model to demonstrate the relationship between data quality and program improvement 
and conclude with a brief overview of the big picture: NRS data flow. We trace how data move 
from the classroom to the federal level and how the U.S. Department of Education needs quality 
data to manage and obtain federal support for adult education. This chapter also supplies 
examples of how various stakeholders use data and provides brief vignettes of programs that 
have become adept at different aspects of collecting accurate data on thousands of students each 
year. 

Data Defined 

Data are Information 

Quite simply, data are valuable because they represent information. Webster’s dictionary 
defines data as “factual material used as a basis for discussion or decision: information.” In an 
adult literacy program, data are information about how your students perform, how long they 
attend and what works in your program. Good data are information that will help answer 
questions you and your staff have about recruitment and retention, the effectiveness of your 
program and its components. Data are a student’s pre- and posttest scores, a student’s goals, the 
number of hours a student attends class or the number of students who drop out. If a program’s 
administrators, teachers or supporters are curious about the types of students who drop out or 
make great or poor gains on tests, they must collect demographic data. These data — age, 
employment status, family status and educational attainment — are crucial to any effort to analyze 
patterns to pinpoint areas of strength and weaknesses. 

Data Often Surprise... And Clarify 

It is not uncommon for the staff of a literacy program to be surprised by what good data 
reveal about their efforts and beliefs. When staff analyze data, such as pre- and posttest results 
or the average number of hours of instruction received by students, there is usually some 
evidence that supports a staff s belief that the students are learning and attending. The numbers 
often show, however, that some types of students are doing better than others, some do not attend 
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frequently, some types of instruction produce better results or the gains students are making are 
not as great as the staff had assumed. 

That this is so is not really a surprise. People who work in helping and teaching 
professions, and literacy programs are often both, are usually optimists who believe in the 
individual. They are also realists who are trying to help people overcome overwhelming odds or 
years of poor, or no, instruction or treatment. As realists, they do not expect everyone to succeed 
and tend, as optimists, to see most clearly the small and large achievements of their students, 
some of which are not academic achievements. The staff is urged to tell the success stories to 
public and private supporters, who for years were satisfied by those stories. Funds continued to 
flow and these funds, and the continued success of individual students, confirmed to program 
staff that they were doing good work. 

The problem is that anecdotes can obfuscate the larger picture. There is a tendency to 
believe that these stories represent the whole picture. In this era of accountability for all 
education programs, anecdotes and personal success stories are no longer enough. They cannot 
tell you, for example, what type and size of growth you can expect and what percent of your 
students should achieve this type of growth. When programs begin to measure results for all 
participants, they are inevitably surprised. Data clarify, as the accompanying vignettes illustrate. 

Quality Data and Program Improvement 

When you discuss how you would answer “would it make a difference” questions and 
other issues, program staff inevitably sees the need for test scores, demographic data, hour of 
attendance summaries and other kinds of information, otherwise known as data! Accurate NRS 
data can help a program improve its enrollment, increase retention and improve student 
outcomes. 

Sometimes just attempting for the first time to collect data leads to program changes and 
improvement. All students want to know if they are making progress. All teachers and program 
managers usually want to know if their program is working as designed or if it needs to be 
redesigned. NRS data can indicate whether students are staying long enough to make gains and 
the types of students who make these gains. An analysis of NRS data can help pinpoint at which 
levels students seem to make the most progress. If the program database has the sophistication, 
it can help teachers understand the types of students who are making the most progress or 
dropping out the soonest. 

Exhibit 2- 1 illustrates the process for using data for program improvement, using 
advancement of beginning ABE students as an example. The process includes reviewing data to 
identify issues or problems to address; developing a data analysis and program improvement 
plan; implementing that plan by adding resources, changing staffing or revising procedures; 
collecting more data; and reviewing data again to assess whether the changes made a difference. 
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“Data Clarify:” Vignettes 

❖ A large literacy program in New York City emphasized writing instruction along 
with reading instruction for low level students. The staff believed that reading and 
writing were linked and that when students become proficient in one skill, they learn 
the other. When the program’s outside evaluator compared the students’ writing 
gains to their reading gains, however, she concluded that gains in writing do not 
necessarily translate into gains in reading or vice-versa. 

❖ The staff of a Montana adult learning center had collected mandated data for years 
but had never used it in a systematic way to assess and improve its program. The 
staff thought it was serving its students well but when it compared three consecutive 
years of data, staff realized that many students were exiting before meeting their 
goal of obtaining a GED. 

❖ A small literacy organization in North Carolina relied on volunteer tutors to provide 
instruction. The program’s staff believed that when students failed to show growth 
in skills on standardized reading tests, it was the tests that were at fault. When the 
staff, however, began to track the actual number of hours of instruction that students 
were receiving over a six to nine month period, they realized there was too little 
duration and intensity of instruction taking place to get growth. 

❖ The staff in two large, experienced literacy programs in two of Pennsylvania’s 
largest cities always believed that their students did not want to be tested. After 
being trained and involved in evaluation techniques, both programs began testing 
some of their students. They quickly learned that the great majority of their 
students — from beginners to GED candidates — not only wanted to be tested but very 
much wanted to know their test scores. The programs’ sites that initially failed to 
share the results with the students soon began to do so. There was some evidence 
that when staff discussed test results with students shortly after the administration of 
the tests, the students were more likely to stay long enough to be postested. 

❖ A Chicago literacy program in a large community-based organization almost lost 
funding due to a community and agency rumor that retention in the GED classes 
was poor. The program manager presented data that proved the rumor untrue. 
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Exhibit 2-1 

Data Quality and Data Use Model 




♦> Review data. The process begins by reviewing data on a regular basis. Data 

provides you with a picture of how your program is working. By critically evaluating 
this picture, you can get clues as to what is not working or your curiosity may be 
raised as to why things are working as they are. For example, why are some students 
posttested and not others? What makes some groups of students stay longer than 
others? Why are students more successful in some classes? 

♦♦♦ Identify issues and problems. Reviewing data is bound to identify program areas 
that seem problematic or are otherwise worthy of further investigation. Y ou should 
select and focus on issues and problems to address in your program improvement 
efforts. 

♦♦♦ Develop questions. It is helpful to treat the issues you want to address as a research 
project. To this end, formulate measurable questions based on your analysis of the 
data. Will changing posttesting procedures result in a higher percentage of students 
being assessed? Will retention and average instructional hours increase if the 
program adopts a managed enrollment approach? It is important to specify a 
measurable outcome that will reflect program improvement efforts. Otherwise you 
will never know whether your changes worked. 

❖ Develop data analysis plan. If you have measurable research questions, you expect 
the changes you make to be evident in your data. In other words, your data will 
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reflect the program improvement efforts you make in your program. You need a plan 
on how you will analyze these data to determine the effects of your changes. Without 
a plan you may become overwhelmed and confused about how to interpret your 
findings — or even fail to collect the information you need to evaluate your changes. 

❖ Program improvement plans. Your data have suggested the need to change 
something in your program, you have made the decision to do so and planned on how 
you will analyze the data. You should also develop a plan to improve your program 
to address these issues. The plan should state clearly the resources, staff and 
procedural changes that you need to change to make a difference. 

❖ Resources, staff and procedures. Depending on what you have decided to improve, 
you may need to devote more resources to the problem, such by increasing funding 
for data collection; changing staffing or by altering the responsibilities of staff; or 
changing data collection procedures. Once you implement these changes, you are 
then ready to collect and review data again to assess the effect of your changes and 
begin the data — program improvement cycle once again. 

Data can inform you by revealing relationships among students and services that promote 
program improvement, but only good, quality data can do so. If your data are collected by staff 
that are indifferent or unskilled at collecting data, if procedures are not well developed or defined 
and your program lacks the forms and database to record the information, your data will be 
suspect. You will have large amounts of missing data, pretests without posttests, incomplete 
attendance data and missing student demographic data. All of the effort and expense to collect 
these data will have been wasted and you will not have data to inform program improvement. 

Data Quality and Use at the Federal Level 

As you consider the role of quality data in improving your own program, it is also 
important to remember the bigger picture and the role your program’s data play in supporting the 
adult education program at the state and national levels. Just as having accurate, high quality 
information is the only way you can make informed decisions about your program, the state and 
federal departments of education need accurate information to understand and manage the adult 
education program and obtain funding for instruction and services. 

State adult education directors, federal staff and literacy advocates use NRS data to report 
to legislators, other state and federal agencies, corporate sponsors and foundations to support 
funding and improve programs. The general public and students also have access to this 
information to evaluate the value of adult education. The source of these data is the local adult 
education program itself. When considered from this view, the importance of quality data 
becomes even more critical. 



Chapter II. Quality Data: Why Good Data Matter 




16 



Improving NRS Data Quality 



Using Data for Program Improvement: Vignettes 

❖ In Montana a local program with multiple funding sources and a recent history of 
collecting data for program improvement purposes was able to work with state 
administrators to expand the data fields that the statewide data collection system 
included. This expansion made it possible for local staff to get answers to their 
own questions about their programs and to generate reports to some other 
supporters. 

❖ A large public literacy system in Arizona had a poor posttest rate. Staff started 
keeping track of student hours of attendance and soon realized they were waiting 
much too long to administer posttests. A New York City program, which 
posttested in June, realized that attendance was lowest in June and so it too had a 
poor posttest rate. Both organizations adjusted and considerably increased the 
number of students they posttested. 

❖ Once a program has its first year or baseline data, it can work to improve its 
numbers and publicize its successes. A Pennsylvania literacy organization 
increased the number of students posttested and the average number of hours of 
instruction each student received for three consecutive years. Analyzing its data 
enabled it to determine the minimum number of hours of instruction students 
needed to start showing progress on standardized tests. 

❖ Pennsylvania ’s Project Equal is the state’s comprehensive project to improve data 
use among local programs. Programs review their data, identify areas for 
improvement based on the data and form a program improvement team. The team 
then develops and implements a program improvement action plan, using the 
indicators of program quality to identify areas to improve. Staff reviews data 
again after implementing the changes. To help programs, there is a self- 
assessment guide and training on using data. The program must submit logs on 
data for decision-making to document the process. Trainers, who give staff 
feedback, review the logs and assist staff at the local program. 



Exhibit 2-2, taken from the NRS Implementation Guidelines, illustrates how NRS data 
flow from the instructional site to the program, the state and finally the federal level. At the local 
program level, each of the program’s instructional sites collects measures from students. During 
the intake process, local staff collects descriptive measures — demographic information, student 
goals and status measures — and conducts an assessment of the student’s educational functioning 
level for placement. During the course of instruction, program staff (typically the teacher) 
provides at least two additional measures about the student: contact hours or attendance and a 
progress assessment or posttest. Other student outcomes attained may also be recorded on the 
update. Local or state policy determine when this update information is collected, which may be 
on an ad hoc basis, or at set times during the year, such as monthly or quarterly. 
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In many states, local programs also collect the core follow-up measures on students. 

These measures include the employment-related measures, measures on placement in 
postsecondary education or training and whether the student received a GED or other secondary 
credential. Local programs must combine all of the measures collected at each instructional site 
into an individual student database system. 

Exhibit 2-2 also shows the movement of NRS data from the local program to the state 
level. Each local program submits its data to the state education agency to enable the state to 
develop a statewide adult education database. The state data submission is often on an annual or 
a quarterly basis, although some states require more frequent submission. This state database 
may contain aggregated data from local programs or it may consist of the individual student 
records from programs. At the end of the program year, states must submit the aggregated data 
in the required NRS data tables to the U.S. Department of Education, which maintains a national 
database. 

One of the main points the exhibit illustrates is that as the data flow from sites to the 
federal level, less and less information is available about relationships in the data that reflect the 
procedures used to collect, verify and report data. Consequently, it becomes increasingly 
difficult to evaluate data quality and identify problems in the data collection process. The least 
information is available at the federal level, which receives only aggregated data tables from the 
end of a long and complex data collection process. There is almost no way to pinpoint problems 
or evaluate data adequately at the federal level. 

Quality must be built into data collection at the source of data — the state and local levels. 
The remainder of this guide focuses on how local programs and states can critically evaluate 
methods for collecting NRS data and improve their policies, procedures and training to produce 
quality data. 
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Chapter III. The Data Collection Process 

There are few endeavors where the expressions “garbage in, garbage out” or “you’ll save 
time by doing it right the first time” are more applicable than they are to data collection. As 
presented in the previous chapter, data are information that your program can use for improving 
services to students, that your state can use to manage and improve the state adult education 
system and fund the best programs, and that the federal government uses to demonstrate the 
value of adult education within the federal system. Good program practices — as well as 
funding — depend on accurate and quality information about your program, represented in your 
data. 



The ability of the information or data you have to meet these needs can help you only to 
the extent they are accurate and reflective of the outcomes and processes they are meant to 
represent. If data are collected haphazardly, without clear procedures and sufficient resources, 
the data may not be interpretable or may be wrong — and you will have wasted your time. In 
more scientific terms, your data collection process needs to produce reliable and valid data if 
they are to be useful to you. 

Data are reliable to the extent that they are collected in the same way, by different people 
at different times. In other words, no matter who collects the data or when they are collected, 
they are collected using the same procedures, implemented in the same way. Data are valid only 
to the extent that they represent what they are intended to represent. For example, if you report 
40 percent of students to have gained a level according to test scores, those test scores if they are 
valid, will accurately convey the score and interpretation intended by the test publisher. 

There are three components to collecting valid and reliable data: (1) a well-planned, 
effective process, (2) resources to implement the process and (3) clearly defined procedures for 
collecting each measure. In this chapter, we discuss the characteristics of good data collection 
procedures and ways to implement them in your program. We present a model data collection 
system and a method for evaluating your data collection process. This presentation also discusses 
the resources needed to make your data collection process work. The next chapter presents in 
detail the procedures for collecting key NRS measures for producing valid and reliable data. 

Model Data Collection Process 

It takes the hard work of many people to collect data — people who know what they are 
supposed to do and why they are doing it — and people who are committed to doing it right. Like 
other processes, data collection requires planning and needs constant attention, oversight and 
fine-tuning, through monitoring, error checking and training. You cannot just set up the process 
and let it run itself, assuming everything will work out as planned. Inevitably, problems and 
unforeseen issues will arise, staff may be resistant or new staff will become involved and 
requirements and circumstances will change. However, with a sound, well planned process, 
sufficient resources and oversight, you can have a data collection system that produces valid and 
reliable data that will assist you in managing and improving your program. 
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While the specifics of individual approaches to data collection vary among programs. 
Exhibit 3-1 presents a model data collection process, beginning with student intake, through 
submission of state and federal reports. This model illustrates the key components of a good 
data collection system and staff roles at each step, which we review here. 



Exhibit 3-1 

Local Data Collection: A Model 
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Intake 

When students enter the program, intake staff collects many NRS measures, including 
age, ethnicity, race and gender. Intake often includes a goal setting process, where students, with 
staff guidance, decide on short- and long-term goals for attending class. If a student selects an 
NRS follow-up goal, the intake process should allow recording of this information. If the 
program uses a follow-up survey, the process should include procedures for informing students 
that they may be contacted after they leave class. Intake staff completes an intake form and 
sends the forms to clerical staff and/or teachers. 

Teachers 

Teachers have a large role in data collection in most programs. They must report student 
attendance or contact time, often assess students, report test scores and sometimes are involved 
in the goal setting process. In addition, since teachers have direct contact with students, they are 
often asked to provide student information that was missing or incorrect at other stages of the 
data collection process. Teachers not only complete forms, but ideally also have a role in 
reviewing data and reports. 

Clerical and Error Checking Staff 

The data collection process results in a high volume of paper — forms, test scores, 
attendance records, surveys — that clerical staff receives and must track. Staff must develop an 
organized system for managing this paper flow. The process includes receiving forms from 
other staff for checking and correcting. Once error checkers correct forms, clerical staff then 
sends forms for data entry. 

Data Entry and Data System 

One or more staff members must enter information from forms into the program’s 
database. Data entry may occur at an instructional site or the program may have a central data 
entry point to which all sites submit their forms for key entry. Programs should have an 
individualized student database that is organized to allow the program to examine relationships 
among student and program variables, attendance and student outcomes. Once forms are keyed, 
data entry staff should review error reports promptly and resolve errors and missing data by 
returning forms to the staff members who collected the problem data. 

Reporting and Error Checking 

An essential feature of the data collection process must be regular and frequent review of 
data entered into the data system. The data system should have pre-programmed error reports 
that allow for a review of inconsistent, out-of-range and missing data. Data entry and clerical 
staff should regularly review these reports and should return them to teachers, intake workers 
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and clerical staff to clarify problems and obtain the missing data. Corrections should then be 
sent to data entry staff so they enter them into the database. 

Program Administrative Review 

The process should include a regular opportunity for the program director and other 
program leaders to review data reports. As the person most responsible, the director may often 
be the only person in the program who can see the “big picture,” and thus brings a different 
perspective to the data review process. This review may raise further questions about data 
integrity, requiring another round of data checking and verification among the staff. The 
program director may share data reports with staff as a means to identify problems, track 
progress and buy staff into the data collection process by demonstrating how data can be used for 
program management and improvement. 

State Data System and Reporting 

All local programs send their data to their state agency for integration into the state data 
system. Some states require submission of individual records, while other states require 
aggregated report tables. The state combines the individual program submissions into a state 
report. As part of the data integration process, state staff may identify errors or inconsistencies 
in local data, initiating another round of data checking, cleaning and data entry by the local 
program. 

Federal Report 

All states send their data to the U.S. Department of Education (ED) annually, using the 
NRS data tables. ED then creates a national report and submits the report to the U.S. Congress 
and uses the data in determining state performance incentives. Prior to creating the national 
report, ED reviews each state’s data tables for errors and inconsistencies and asks for corrected 
data tables from states, as needed. In turn, states may once again need to review local program 
data to correct data problems and contact local program directors for corrections. Local staff 
then needs to identify problems and correct errors and resubmit data to the state, which then 
provides corrected tables to ED. 

Summary: Two Key Concepts 

The discussion around the model data collection process, as represented in Exhibit 3-1, 
identifies two key characteristics central to the success of a good data collection system. First, 
the process requires many people to work together as a team. Each point of the process 
represents a staff person who has a definite role in data collection. Each person must know his 
or her job and do it right. Ideally each staff member will also accept responsibility, as a member 
of the team, for fulfilling his or her role. The team makes the process work, which includes 
collecting and recording accurate and timely information, submitting the information to the next 
staff person in the process and reviewing and correcting information that is missed or erroneous. 
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The second characteristic of a good data collection process is that it is iterative, with 
many checkpoints and feedback loops. There are frequent checks on the data — when forms are 
first completed, after data entry, prior to report submissions — and several opportunities to 
improve data integrity. At each checkpoint, there is a staff member who has the responsibility 
and authority to correct the data. In addition, several different levels of staff review the data — 
clerical and data entry staff, teachers, program directors, state and federal staff. This iteration 
and review by staff internal and external to the process produces quality data. 

Improving a Data Collection System 

A serious mistake many programs make is to set up the structure of a data collection 
process — including forms and a database system, provide a quick training to staff, and then 
assume the process will just run itself. As many programs and states quickly learn, this approach 
to data collection is doomed to failure. Yet once you discover problems, it may seem to be an 
overwhelming task to change your system. 

A data process that collects quality data is well planned, has staff members that are well 
trained and who know their roles and responsibilities, and have needed tools and resources. A 
quality system also has oversight to monitor its operation and identify problems promptly. Here 
we review the processes needed to implement or improve a successful data collection process, as 
demonstrated in Exhibit 3-1, that is likely to produce valid and reliable data for your program. 

Collaborative Planning 

Obtaining accurate data is possible when you plan well and motivate your staff to want to 
collect it. You have to build staff motivation and interest to get the buy-in that produces quality 
data — and obtaining buy-in requires collaborative planning. 

A planning group that involves all types and levels of staff will begin the buy- in process. 
The members of the planning group must understand what the NRS requires and formulate what 
assessment or program improvement questions the staff may have. Other planning steps include: 

❖ Develop a clear, written rationale for data collection that addresses what data are to 
be collected and for what purpose, reports that are needed and confidentiality issues. 
Use the plan to build consensus in the organization. 

❖ Clearly describe data collection procedures, including roles and responsibilities for all 
staff. 

❖ Specify error- checking procedures and which staff will perform them. 

❖ Create a schedule to conduct periodic checks of the accuracy of the data. 

❖ Set deadlines for each step in the devebpment process of the data collection system. 
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Your plan should also recognize that it will take time— often two years — to get a system 
that works well. 

Context — Why and for What Purpose? 

It is critical that staff knows what data are collected and how data will be used. An 
attitude such as “because we have to for our grant” is insufficient and will not result in quality 
data. Consequently, your program should develop well thought out plans on how you will use 
data. Consulting with staff as you develop these plans will improve interest further. When staff 
members know that data are used for a meaningful purpose — to improve instruction to students, 
for example — they are likely to take more care in the data collection process. 

Holding regular, interactive staff meetings and training on how data are used will 
reinforce staff buy-in and help develop the skills staff needs to collect good data and interpret it. 
These activities are the best way to deal with the “show me it will make a difference” attitude 
that is often a barrier to institutionalizing data collection procedures. 

Staff Knowledge 

All staff members must know their role in data collection and the specific procedures 
they are to follow to perform their part. You cannot assume procedures are so transparent that 
staff will implicitly know what to do, especially when there is a problem. To ensure staff is 
knowledgeable about the process, you should have clearly defined roles with written job 
responsibilities, written procedures and regular training and interaction of staff on data issues. 
When staff knows clearly its data collection job, you will be unlikely to hear complaints such as 
“that’s not my job!” or “I didn’t know I was supposed to do that.” 

Resources 

Staff inexperienced in data collection often is surprised at the high level of effort the job 
entails. Consequently, a common mistake is to allocate insufficient resources to the task. Staff 
members may be asked to find missing data when they “have the time,” teachers may have to 
key data after class and the program director may review data reports on the weekend, for 
example. While these approaches may not be uncommon, they are likely to produce serious 
shortcomings in data quality. The only way to get good data is to make data collection a priority, 
by providing adequate resources in the form of staff, time and money to the task. Other, tangible 
resources your system needs are forms and a database system. 

Monitoring and Oversight 

A data collection system is like a living process, requiring perpetual care and attention. 
However, like other processes, problems arise: staff changes, unforeseen events occur and 
procedures do not work out as planned. Yet, you will never know whether procedures are 
working unless you build into the process regular oversight and monitoring of data collection. 
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A data collection system with valid and reliable data has several opportunities for staff to 
reflect and revise procedures. Regular formal and informal staff meetings are a good way to 
discover what is working, as is observation of staff members as they collect data. Frequent 
reviews and checks of data soon after data entry also provide an opportunity to ensure data are 
correct. 



Summary 

Since data collection is tedious and staff often misunderstand or do not know how data 
are used, there is often a high level of resistance to data collection. The foregoing presentation 
summarized processes fundamental to designing a data collection process that will result in 
quality data. Many of the processes will also help address staff concerns and complaints about 
the data collection process. Exhibit 3-2 lists some common issues raised by staff and the process 
that will help address these concerns. 



Exhibit 3-2 

Resolving Staff Concerns and Complaints About Data Collection 



Complaint or Concern 


Resolution 


“What are we supposed to do? How is this going to 
work?" 

‘Why doesn’t someone ask me— 1 know what the 
problems are?” 


Engage in collaborative planning to work through 
the process and get staff buy-in. 


“What good is doing this? How is it going to make a 
difference?” 

“What does it matter ifl fill this out right— it’s just a 
dumb report to the state. " 


Explain context and purpose of data so that staff 
knows how it is used and its connection to program 
improvement. 


‘It's not my job— he's supposed to do that. ” 


Provide knowledge and training on roles and 
procedures to avoid confusion about who is 
supposed to do what. 


“ I’m a teacher, not a test administrator! 1 don’t have 
time to do this!” 


Provide sufficient time and resources for data 
collection so that staff have time and take 
responsibility for data collection. 


“1 missed the training— what am 1 suppose to do 
with this? Oh well, I’ll just make something up. ” 


Conduct regular monitoring and oversight to 

identify problems on an ongoing basis. 



The next section provides a systematic way for you to review your data collection 
processes, using the principles presented in this chapter. 
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Evaluating Your Data Collection Process 

Your program already has some system for data collection in place and you may have 
already implemented many or all of the support systems needed to get it started and keep it 
going. However, a process like data collection is never perfect and most likely, there are some 
areas that you do well and others that need improvement. The evaluation form at the end of the 
chapter will help you analyze and assess your data collection procedure and identify areas for 
improvement. 

Data Collection Policies and Procedures 

The checklist on the evaluation form provides a systematic way for understanding the 
data collection process. The annotation below explains each point in the checklist. 

1. Staff has a clear written description and understanding of its roles and 
responsibilities for data collection. 

Like it or not, collecting data now involves every staff member in an adult education 
program. Intake staff may collect student demographic data and goals; teachers report 
attendance, may administer tests and report other outcomes; administrators must 
review and make decisions based on data tables; and administrative staff may be 
involved in checking forms and data entry. Your program should have a clear written 
description of your data collection process and the role of every individual in that 
process — and every program staff member should know his or her role. In fact, job 
descriptions for all staff should incorporate the data collection responsibilities of the 
job and performance reviews should consider how well staff fulfilled these functions. 

2. Clear definitions for each measure have been established. 



Just as it is essential that all staff members know their role in data collection, it is 
critical that they have an understanding of what they need to collect. Your 
procedures should include a written, precise definition for each data item that is 
compatible with your state’s definitions. Some programs and states, for example, 
have a data dictionary that defines all measures and categories within measures. 

While some measures may seem straightforward — ethnicity or sex, for example- 
others, such as student goal setting, may require detailed explanation. However, even 
seemingly simple definitions can sometimes need elaboration. For example, it should 
be clear to staff about how to classify the ethnicity of a student who claims to be part 
Asian and part White. Such ambiguities make it helpful to customize definitions to 
your particular circumstances and to include examples of how to resolve ambiguities. 
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3. Program uses standard forms, tied to the program database, for collecting data. 



Staff must record information on intake and other data forms and administrative staff 
then keys the information from these forms into the program database. Consequently, 
your program should use standard forms for data collection that inc lude all the data 
elements and categories your database system uses. Staff should not need, nor be 
allowed, to enter their own codes or variables, as this will cause data entry errors and 
hurt reliability and validity. Some states with uniform state database systems have 
standard forms used statewide for this reason. 

4. Program has an error-checking and quality control system for identifying 
missing and inaccurate data. 



In an activity as complex and stressful as data collection, mistakes and missing data 
are inevitable. Staff may fail to complete forms fully due to high workload or simple 
oversight. In addition, the required information may not be available when it is 
needed. All good data collection systems have procedures for checking data for 
completeness and accuracy at several times during the process. Data checking should 
follow a regular, prescribed schedule with clear deadlines. It is also a good idea to 
assign one or more staff persons to perform these data checking functions explicitly 
and make the job of this person known to all other staff in the program. Data 
checkers should review all data forms as soon as possible for completeness and 
accuracy and should receive error reports from the database to check immediately 
after data entry. To do their job, data checkers must have access to all staff — 
teachers, intake staff, counselors and administrative staff — and the authority to obtain 
cooperation from them. 

5. Program has ongoing training on data collection. 



The best data collection procedures do not result in valid and reliable data unless staff 
understands and follows them. Therefore, part of your data collection process must 
include training of all data collection staff on their roles and responsibilities, as well 
as the importance of data collection. You should provide this training to all staff and 
it should be offered several times during the year, if possible, to accommodate new 
staff and to allow existing staff to take follow-up training. It is also a good idea to 
have regular meetings or in-service trainings on data issues to give staff the 
opportunities to discuss problems and issues that arise. By addressing these issues 
promptly, you will avoid more serious data problems later. (See the training sections 
in Chapter IV and Chapter V for additional guidance on providing training on data 
collection.) 
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6. Program has a student -level, relational database system. 

To allow you to use data for program improvement, you will need the ability to look 
at student outcomes and demographics by student according to such variables as the 
number of instructional hours received, length of time of enrollment, the teachers and 
classes enrolled and by student educational functioning level. This type of analysis 
requires a database system that stores info rmation by individual students and allows 
you to link the different pieces of data for each student in reports or other output, a 
system known technically as a relational database. Your state may have a statewide 
data system that you use that meets these requirements. If not, your program should 
invest in such a system to enhance the value and use of the data for you and to allow 
you to produce required NRS and other reports more easily. 

7. Data entry procedures are clear and timely. 

Your procedures for data entry should specify at least one person whose job is to 
enter the information from forms into the program’s database. All staff members 
should know this person’s role and he or she should have the authority to request 
clarification or resolution of errors. In addition, data entry should be scheduled at 
frequent, regular intervals — such as weekly or monthly. Without frequent data entry, 
not only will you have a large backlog of forms to enter, but you may not become 
aware of errors and missing data on forms until it is too late to correct them. Part of 
the data entry procedures should also include a prompt, organized way to identify and 
resolve errors. For example, you should print error reports from your database as 
soon as possible after data are entered and staff should correct errors and resolve 
missing data as soon as possible after data entry. 

8. Staff has timely or direct access to information from the database. 

A frequent complaint of staff involved in data collection is that they fill out the forms, 
enter the data and then never see it again. If staff members cannot access data, they 
cannot learn how data are used or how they themselves can use it for program 
improvement and management. Your database system should have the capability for 
local program staff to access their data in useful ways. It is best if this access is 
direct, so that staff at the local level can query the database to print a report locally. 
Access through a third party or through the state will also work for local programs, if 
they are able to receive the data they request in a timely fashion. The usefulness of 
the data is limited if it takes weeks or months for program staff to receive data tables. 

9. Staff regularly reviews data. 

Your data collection procedures should include a regular review of data soon after 
entry into your database. Regular data reviews allow you to spot errors, missing data 
and other data that “don’t make sense.” You can use data reviews as a staff 
development opportunity to examine problems and issues to support program 



24 



Chapter III . The Data Collection Process 



Improving NRS Data Quality 



improvement. Data can help you and your staff understand issues such as the impact 
of instructional arrangements, learner retention and learner progress. Having your 
staff learn to use data may not only foster program improvement, but may improve 
data quality, as staff sees the importance of data collection to produce valuable 
information. 

Indicators of Performance 

Data collection is a complex process, involving many staff. It thus requires regular 
review and monitoring. You may identify issues to explore further after completing the checklist 
or reviewing the flow chart in Exhibit 3-1. There are several ways to investigate your data 
collection process in depth, including discussing processes with staff that is involved in data 
collection and observing staff members as they do their job. Your program’s data can also give 
you clues about needed improvements to data collection. 

Discuss with staff. Talk to all staff members about their understanding of the data 
collection process to ensure they are clear about what they are doing and the reasons for 
procedures. You might also ask them for insight into what is not working and what they think 
they could do better. Through this discussion you might also explore areas of resistance among 
staff and reasons why procedures are not working as they should. Staff may also have insights 
into ways to resolve data collection problems. You could use this information to work with staff 
to identify a plan for improving data collection procedures. 

Observe data collection. Observe staff most directly involved in data collection to 
evaluate whether processes are working as you envision them. For example, observe how data 
are entered, forms completed and missing or erroneous data are resolved. Identify areas of 
strength and weakness and discuss with staff. 

Review program data. Review your program’s data for anomalies, missing data and 
other inconsistencies to get insight into areas where more training may be needed. You should 
examine data tables and look at data entered or collected by individual staff members. For 
example, you could: 

❖ Spot-check data in your database against original data on forms. 

❖ Check for out-of-range values and look for numbers greater than 100% and number 
of students in sub-samples greater than total enrollments. 

❖ Examine individual student records for missing data and highest and lowest values 
recorded, to see whether they make sense. Check whether it is possible to obtain these 
values or scores. 

❖ Examine the above data by individual staff member involved in completing forms or 
data entry. 

❖ Examine data again after it has been checked and completed. 
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♦> Before you submit them, examine NRS and other accountability reports for missing 
data, out-of range values and patterns that seem suspect (e.g., advancements with too 
few hours, too high or too low percentages of outcomes achieved). 

When you evaluate these data, consider whether your program’s data collection policies and 
procedures are clear and understandable. If training is not the problem, you need to revisit or 
better define procedures. 



26 



Chapter III. The Data Collection Process 



Improving NRS Data Quality 



Case Study #1 : Local Program Data Collection 

Staff at Literacy Volunteers of America-Chippewa Valley (LVA-CV) located in Eau 
Claire, Wisconsin, found they were collecting too much data — and they had difficulty organizing 
and using them. At year’s end, they found themselves floundering in long, detailed reports from 
ten individual programs spread out over three counties. What was worse, all were using slightly 
different recording systems to collect data. This made it particularly challenging to compile 
program and organization-wide evaluations and to analyze the data. The sheer quantity of data 
was obscuring the essential information and impeding their progress and ability to share 
successes and challenges of the students served in their programs. LVA-CV realized they 
needed to develop a clear and concise data plan that would allow meaningful articulation of its 
successes and challenges. 

The challenge was to pull consistent pieces of information from all segments, record that 
standardized data accurately in a computerized collection system, consolidate the findings, and 
produce a report. Staff understood that by first determining what information they needed, they 
would avoid a lot of wasted time and energy. 

LVA-CV developed an efficient and standardized data plan, involving a cycle of 
collecting, analyzing, organizing, revising, and articulating: 

• Examine your organization’s strategic plan; 

• Determine the questions you need to ask; 

• Develop/revise the data plan: 

? Define the roles, 

? Establish timelines for assessment process, 

? Standardize data collection process, 

? Revise forms for consistent usage, 

? Review testing practices and standardize, 

? Strengthen staff communication; 

• Aggregate data for reporting; and 

• Analyze your data to tell your story. 

Although, the staff recognized they would never achieve a perfect system, they realized 
that efficient standardized data collection was essential to continuous program improvement. 

Contact: Heidi L. Fisher, Carol L. G abler 
Literacy Volunteers of America-Chippewa Valley 
http:/ /wwwjyacv. ore 

Adapted from Adventures in Assessment, Volume 13 (Spring 2001). 

SABES/World Education, Boston, MA Copyright 2001. 

Posted to SABES Web Site: August 2001: 
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Case Study #2: Local Program Data Collection 

The Albany Park Community Center, a community based, non-profit organization in 
Chicago that provides a range of social services, wanted to consolidate the multiple data 
reporting forms required by its adult literacy funders and develop a more user- friendly reporting 
system for the program. Due to problems with databases and data entry, important data were not 
being combined, compiled, or compared. For example, program administrators could not readily 
correlate test gains with family literacy outcomes because the databases operated on different 
platforms making it difficult to import and export key data. In addition, data on learners’ native 
languages and countries, their educational goals, and their satisfaction with the program had 
never been entered into any computer database because no funder had asked for this data. 

The organization realized that it needed to approach program reporting and data 
collection more systematically. The first step was to develop a plan with all staff members by 
discussing: 

• What to assess (English literacy skills, learner goal attainment, learner satisfaction or 
persistence)? 

• What instruments to use (TABE, CASAS, locally made performance assessment)? 

• What kind of data to collect (previous schooling, goals, parent, employment status)? 

• When and by whom would the assessments be administered? 

Next program administrators identified the staffing expertise and the types of talents 
available inside and outside the program. These included: 

• Planners’, program administrators, teachers, students, top management (beyond the 
literacy programs); 

• Database design: a database consultant; 

• Data entry, support staff, student volunteers; 

• Statistical compilation and analysis : program administrators, consultants; and 

• Interpretation and application of results’, teachers, students, other staff, top 
management. 

After a series of meetings over 1 8 months and experimentation with different databases, 
staff produced the following work plan. 

• Seek input from staff and students on the purposes and uses of evaluation. 

• Design of a comprehensive database that can import and export data with some of the 
funders’ reporting system, produces needed table and has a query function. 

• Train support staff and program coordinator in use of the database — data entry, forms 
and queries. 

• Involve teachers, students and managers in interpreting the results. 

Contact: Jeffrey P. Bright 
Albany Park Community Center 
http://www. albanyparkcommunitycenter. org 
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Sample Data Table #1 

Data Collection: Core Follow-Up Outcome Of Obtained A GED Or 

Secondary School Diploma 



Program 


Number of 
Participants 
with Goal 


Number of 
Participants in 
Followup 
Sample 


Response 

Rate 


Number of 
Participants 
Achieving 
Outcome 


Percent of 
Participants 
with goal 
Achieving 
Outcome 


Program 1 


4,230 


4,230 


100% 


2,289 


14% 


Program 2 


2,233 


0 


0% 


1,291 


58% 


Program 3 


6,300 


0 


0% 


1,735 


28% 


Program 4 


10,162 


8,494 


81% 


5,283 


52% 


Program 5 


28,525 


0 


0% 


7,609 


27% 


Program 6 


6,704 


0 


0% 


1,665 


25% 


Program 7 


2,135 


2,135 


100% 


1,623 


76% 


Program 8 


948 


948 


100% 


349 


37% 


Program 9 


281 


304 


108% 


74 


26% 


Program 10 


135,683 


0 


99% 


34,981 


26% 


Program 1 1 


15,653 


9,710 


59% 


4,424 


28% 


Program 12 


2,359 


1,440 


53% 


2,459 


104% 


Program 13 


4,965 


0 


0% 


1,828 


37% 


Program 14 


15,331 


NA 


100% 


2300 


15% 


Program 15 


13,674 


13,674 


54% 


6,248 


46% 



Potential Problems 

• Program 10 reports that 0 participants were included in the study, but reports a 99% 
response rate. 

• Program 1 has reported an error, 14% of 4230 is not 2289. 

• Program 9 has reported more participants responding at follow-up than as having the 
goal initially. 108% is an obvious mistake. 

• Program 12 has more students achieving the goal then there are students with the 
goal. 

• Program 14 reports a low percentage of students with goals achieving the goal. This 
percentage may reflect a data entry mistake, a problem with data collection processes, 
or may indicate a poorly functioning program. 
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Sample Data Table #2 

Data Collection: Raw Data Table by Student 



Student 


Pretest 

Score 


Posttest 

Score 


Gender 


Ethnicity 


Age 


Carvajal 


196 


201 


1 


1 


19 


Swaminathan 


256 


259 


2 


5 


87 


Abul 


211 


192 


2 


5 


24 


Rios 


184 


186 


9 


1 


26 


Sanchez 


136 


152 


1 


1 


45 


Martinez 


237 


246 


2 


1 


31 


Fabio 


220 


231 


1 


11 


37 



Potential Problems 



• There is a large drop in Abul’s score between pre and posttest. This drop is possible 
but could indicate data collection or data entry error. 

• Gender is miscoded for Rios; this should be checked with the data collector. 

• Eighty- seven is an extreme value for the age of a student, may be data entry error. 

• Ethnicity is miscoded for Fabio, ‘11’ is an out-of-range value. 
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Data Collection Procedures Evaluation 

A, Describe Your Data Collection Procedures 

Describe your program’s entire data collection process beginning at student intake 
through submission of state reports. Describe who collects data at each point, how they are 
collected, data entry procedures and error checking. Also include how your reports are prepared, 
submitted and reviewed. Then complete the checklist in Part B. 

B. Rate Your Data Collection Procedures 



Data Collection Procedures Checklist 


Have in 
place 


Have, but 
need 

resources 


Have, 

needs 

change 


Do not 
have 


1. 


Staff has a clear description and understanding of its roles and 
responsibilities for data collection. 










2. 


Clear definitions for each measure have been established. 










3. 


Program uses standard forms, tied to the program database, for 
collecting data. 










4. 


Program has an error-checking and quality control system for 
identifying missing and inaccurate data. 










5. 


Program has ongoing training on data collection. 










6. 


Program has a student-level, relational database system. 










7. 


Data entry procedures are clear and timely. 










8. 


Staff has timely or d irect access to information from the 
database. 










9. 


Staff regularly reviews data. 











C. Evaluate Your Data Collection Procedures 



Identify the aspects of your program’s data collection procedures that you are not 
performing well. Describe your areas of strength and weakness, by reviewing the areas you 
rated as “needs change” and “do not have.” If you checked any areas as “need resources,” 
determine what resources you are lacking and how you might get them. Also, review the 
accuracy and completeness of your description of data collection in Part A on the form. After 
discussing issues with staff, develop a plan for improving weaknesses. 
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Areas of Misunderstanding or Lack of Knowledge 



Areas of Strength 



Areas of Weakness 



Plan for Improvement 



Problem 


Plan for 

Correcting Problem 


People 
Responsible 
for Correcting 


Timeline for 
Completing 
Corrections 


1 . 








2. 








3. 
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Chapter IV. Collecting NRS Outcome Measures 

The heart of the NRS is its outcome measures: educational gain, enter and retain 
employment, obtain a secondary credential, and enter postsecondary education. States set 
performance standards for these measures and program effectiveness is judged in part by 
whether these standards are met. These outcome measures are also a set o f indicators that tell 
you how well students are doing in your program. They can help you evaluate what is working 
in your program — and what might need improvement. Consequently, collecting these measures 
in valid and reliable ways is vital to your program. 

In this chapter, we focus on the NRS procedures for collecting educational gain and 
follow-up measures. For each measure, we first present an overview of the measurement 
procedures within the NRS framework, and then provide a step-by-step explanation of how to 
collect it. Tied to each measure is a checklist and evaluation form you can use as a self- 
assessment tool to assess how well your program performs each of the data collection steps. 

Each section provides suggestions on methods you can use to evaluate your programs’ 
procedures and includes sample data tables to exemplify how you can use data as part of the 
evaluation process. There is also a case study in each section that provides a concrete example 
of how to implement the procedures discussed. 

This section also includes a discussion and self-evaluation of intake and goal setting and 
local staff professional development. Effective goal setting is central to the collection of the 
NRS follow-up outcome data, while the foundation of any data collection is strong staff training 
in the process. Without effective professional development, your program cannot implement any 
data collection procedures, no matter how well designed. 



Assessment: Measuring Educational Gain 

Many program directors, teachers and policymakers believe that the adult education 
program is first and foremost an educational program designed to teach literacy skills. The 
Workforce Investment Act affirms this view by explicitly defining student literacy gains as a key 
outcome measure of the program. For this reason, many adult educators see educational gain as 
the most important outcome measure within the NRS, since it is the measure of student 
acquisition of literacy skills. 

The measure is defined according to NRS educational functioning levels, which are 
determined according to the student’s basic or functional skill level. The program assesses the 
student upon entry into class and places the student into an initial level, according to NRS 
definitions. After receiving instruction, a student is again assessed at a time according to state 
policy to determine educational level. If according to the posttest, the student is at a higher level, 
the student is reported as having advanced an educational functioning level. 

There are six educational functioning levels each for ABE and ESL. For each level, there 
is a set of descriptors that describe student entry level skills in the areas of reading and writing, 
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numeracy and functional skills for ABE. There are also speaking and listening descriptors for 
ESL. Each level has a set of benchmarks of commonly used tests as a guide for placement. 

The accurate placement of students into educational functioning levels, both initially and 
upon posttesting, is critical. Such placement is not only good instructional practice, but allows 
for an accurate measure of educational gain. 

Procedures and Policies 

To measure educational gain, your program should have the following policies and 
procedures in place. Use the assessment evaluation form and checklist to rate your program in 
each area. 

1. Appropriate assessments selected, following state requirements. 

NRS guidelines require that states select the assessments for all local programs in the 
state to use. The assessment must either be a standardized test or an alternative 
assessment procedure that uses standardized scoring rubrics that are valid according 
to accepted psychometric standards. Many states allow more than one assessment. 
For example, CASAS or BEST is often used for ESL students, and the TABE is 
commonly used for ABE students. Some states may also require different 
assessments for measuring reading and writing. Your program must use the 
appropriate state required assessment. 

2. Appropriate form or version of the assessment. 

Assessments designed for multiple administrations on the same students, such as for 
pre- and posttesting, have different, but equivalent versions or forms. The different 
forms are usually designated by numbers or letters. For example, the TABE has 
forms 7 and 8; the BEST, a Form A and Form B. A different form must be used for 
pre- and posttesting, so that students do not remember the items from the previous 
administration. In addition, some tests such as the TABE have different forms for 
student proficiency levels, designated as “easy” and “hard,” for example. When 
using such a test, programs must follow the test publisher’s guidelines in selecting the 
correct test form for each student. 

3. Established time to administer the initial assessment. 

The initial assessment is the basis for placing students in an entering educational 
functioning level according to NRS or state definitions. It is the baseline upon which 
programs measure student learning gains. Programs should administer the initial 
assessment to students at a uniform time shortly after enrollment. This time may be 
set by state policy or by the local program. It is advisable not to administer the 
assessment during the first few days of class, but within a few weeks of enrollment. 
Regardless of the date for initial assessment, it should be uniformly applied to all 
students to make test results comparable across students. 
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4. Procedures for placing students in the appropriate level, based on the results of 
the initial assessment. 

Using the results of the initial assessment, programs should place students into the 
appropriate NRS educational functioning level or the equivalent state level. The NRS 
Implementation Guidelines clearly define the criteria for entry into each level, using 
test scores or the educational level descriptors. Programs do not need to use all of the 
areas described in the level descriptors to place students, but should use the areas 
most relevant to the student’s needs and program’s curriculum. For example, if the 
student’s goal is to improve reading skills, the reading and writing descriptors should 
serve as the basis for placement. However, if multiple skill areas are assessed and the 
student has differing abilities in each area, NRS policy requires that the program 
place the student according to the lowest skill area. 

5. Established uniform time to administer the posttest or follow-up assessment. 

Just as you must administer the initial assessment to students at a uniform time, you 
must administer the follow-up assessment or posttest to students at a set time. This 
uniformity allows for comparability of tests scores across programs. NRS guidelines 
require your state to set the time for posttesting for the state, which your program is to 
follow. This time may be after a set number of instructional hours or months of 
instruction and should be long enough after the pretest to allow the test to measure 
gains. As noted under point number two, you must posttest with the parallel form of 
the same assessment as used to place the student. 

6. Procedures for determining level advancement of students based on the posttest or 
follow-up assessment. 

Educational gain, the literacy measure reported for the NRS, is determined by 
comparing the student’s initial educational functioning level with the educational 
functioning level determined through the follow-up assessment or posttest. In 
determining gain, you must use the NRS Guidelines definitions, which use test scores 
and skill descriptors. If your state has other, equivalent criteria, use them to 
determine educational gain. For example, if a student initially scored 510 in reading 
on TABE 7 and later scored 550 in reading on TABE 8, the student would advance 
one level, from ABE low intermediate to high intermediate ABE. Likewise, an oral 
BEST score from 25 initially to 45 on posttesting moves the student from beginning 
ESL to low intermediate ESL. It is important to note that if a student is not 
posttested, no advancement can be determined for that student. The student must 
remain in the same level as initially placed for NRS reporting. 

7. Staff trained in administration of the assessment, including timing of assessment, 
scoring and providing feedback to students. 

The results from a standardized test are only meaningful if staff follows the 
procedures for proper administration of the assessment. These procedures include the 
steps outlined above — i.e., use of the correct form of the assessment and 
administration at the proper time — and also include following the publisher’s 
procedures for giving directions to students, timing of the assessment and not 
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providing help to students. Assessments should also be administered under good 
conditions — in a well- lighted, quiet room, for example. It is advisable that staff other 
than the student’s teacher administer the assessment. Your program must provide 
training to all staff who administer tests on the correct administration procedures. 
Such training should be provided on an ongoing basis to accommodate new staff and 
as a refresher to staff who had earlier training. 

Indicators of Performance 

If you uncover problems with your program’s assessment procedures, or you want to 
investigate procedures, you can discuss procedures with staff and observe staff, review student 
records or examine your program’s data. The following methods give you a good indication of 
how well staff has followed procedures and also lets you evaluate whether your data “make 
sense.” 



Discuss with staff. Have each staff member involved in the student assessment process 
complete the assessment evaluation form. Discuss the results with them and identify a plan for 
improving areas of weakness. 

Observe staff. Observe staff conducting assessments, using the evaluation form during 
your observations. Identify areas of strength and weakness and discuss with staff. 

Review a sample of student records. Select a sample of your program’s student records 
to determine whether assessments are conducted properly, such as whether staff used the 
appropriate test and form and when the test was administered. 

Review program data. Review the assessment data of your program overall to evaluate 
whether it provides a clear and accurate picture of your students’ performance on the 
assessments. Anomalies or things that “don’t make sense” give you clues to what might not be 
working correctly. For example, all students would not usually gain a level. Other data to 
examine include: 

❖ The number and percentage of students pre- and posttested. 

❖ The number of students placed in each level. 

❖ The percent of students advancing by level. 

❖ The average number of instructional hours it takes students to advance by level. 

❖ The average number of hours of instruction received by level. 

Look for student groups that have higher or lower than average numbers — that stand out 
compared to the others. Try to determine why these anomalies exist. 
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Case Study #3: Measuring Educational Gain 

The Ysleta Community Learning Center is located near El Paso, Texas, about half a mile 
from the US/Mexico border. At the Center, awareness on the part ofteachers, students, 
administrators and staff that their progress is being monitored has created an environment of 
accountability and a focus on instruction. 

In the past, students could advance from one educational functioning level to another on 
the judgment of the teacher or request from the student. With the implementation of 
performance- based funding in Texas, as well as the NRS, students now can only move from one 
level to another based on their pre- and posttest scores. Center staff assesses students after every 
40 hours of instruction, allowing students to progress in the mid-term, as well as at the end of the 
semester or program year. Program staff has found that using exams to move students forward 
helps staff with student goal setting and to pinpoint areas of growth, and makes students and staff 
more accountable. 

An important component of the assessment process is feedback to staff. The ABE 
director obtains the data from ACES, the state data system, and provides teachers with charts that 
compare their classes to similar classes by time and subject. Staff also compares contact hours, 
percent progress tested, completion of one or more levels, and GED achievement. The data are 
then discussed as a group and individually with the director. Based on the data, goals for the 
coming semesters are set. Since the assessments help teachers gauge student successes and what 
works, they are eager to ensure that students are progress tested and they are very attentive to 
issues of retention and attendance. Going over the data has given teachers ownership of the 
process, which helps ensure progress testing gets done. 

Contact: Irma H. Trujillo 
Ysleta Community Learning Center 

http://www.ysleta.isd.tenet.edu/html/ysleta_community_learning_cent.html 
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Sample Data Table #3 

Educational Gain: Percentage of Students Pre- and Posttested by ABE Level 



ABE Level 


N 


Percent 

Pretested 


N 


Percent 

Posttested 


Beginning Literacy 


59 


92% 


43 


40% 


Beginning Basic 


12 


95% 


6 


29% 


Low Intermediate 


55 


84% 


17 


34% 


High Intermediate 


71 


13% 


34 


18% 


Low ASE 


64 


81% 


32 


38% 


High ASE 


47 


79% 


79 


41% 



Potential Problems 

• There are significantly fewer students posttested in the Beginning Basic Level than at 
other levels. This percentage should be checked and if accurate, indicates assessment 
practices should be improved so that more students are tested. 

• The percent of High Intermediate students who were tested (both pre- and post-) is 
lower than for the other levels. Also, more students were posttested than pretested. 
These anomalies may indicate a testing problem. 



Sample Data Table #4 

Educational Gain: Percent and Number of Students Advancing and Hours to 

Advance by Level 



ABE Level 


N 


Number 

Advancing 


Percent 

Advancing 


Average 
Hours to 
Advance 


Beginning Literacy 


36 


31 


85% 


87 


Beginning Basic 


7 


5 


76% 


42 


Low Intermediate 


24 


17 


72% 


12 


High Intermediate 


21 


10 


45% 


41 


Low ASE 


19 


18 


95% 


39 


High ASE 


17 


13 


78% 


37 



Potential Problems 

• Significantly fewer students in the High Intermediate Level advanced and many more 
in the Low ASE Level advanced. This could be a sign of instructional problems with 
students in these levels or data entry errors. 

• The number of hours needed to advance for the Low Intermediate Level is low 
compared to the others and should be checked. 

• Advancement rates overall seem high and should be verified. 
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Assessment Procedure Evaluation 

A. Describe Your Assessment Policies and Procedures 

Describe your program’s assessment policies and procedures. Include in your description 
the assessments and its forms used for ABE, ESL and ASE, when pre- and posttesting takes 
place and training provided to staff. Then complete the checklist in Part B. 

B. Rate Your Assessment Policies and Procedures 



Assessment Checklist 


Have in 
place 


Have, 
but need 
resources 


Have, 

needs 

change 


Do 

not 

have 


1 . Appropriate assessments selected, following state 
requirements. 










2. Appropriate forms or versions of the assessment, using 
different forms at each administration. 










3. Established uniform time to administer the initial assessment. 










4. Procedures for placing students in the appropriate level based 
on the initial assessment. 










5. Established uniform time to administer the posttest or follow- 
up assessments. 










6. Procedures for determining level advancement of students 
based on the posttest or follow-up assessment. 










7. Staff trained in administration of the test, including timing of 
test, scoring and providing feedback to students. 











C. Evaluate Your Assessment Policies and Procedures 

Note aspects of assessment procedures you or your staff misunderstand or lack 
knowledge about. Describe your program’s areas of strength and weakness in assessment, by 
reviewing the areas you rated as “needs change” and “do not have.” If you checked any areas as 
“need resources,” determine what resources you are lacking and how you might get them. Also, 
review the accuracy and completeness of your description of procedures in Part A on the form. 
After discussing issues with staff, develop a plan for improving weaknesses. 

Areas of Misunderstanding or Lack of Knowledge 
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Areas of Strength 



Areas of Weakness 



Plan for Improvement 



Problem 


Plan for 

Correcting Problem 


People 
Responsible 
for Correcting 


Timeline for 
Completi ng 
Corrections 


1 . 








2. 








3. 
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Intake and Goal Setting 

Setting students’ goals is an integral part of the education process. Establishing goals not 
only helps define the areas in which instruction and learning is to be focused, but goals also 
provide a benchmark by which programs and students can assess their progress. For these 
reasons, the goals set should be attainable and measurable. 

Within the NRS framework, all students are assumed to have at least one goal: 
development of literacy skills. That is, all students are assumed to be in the program to improve 
their literacy skills. This assumed goal is the reason that all students are counted in the 
educational gain measure. Students often have other goals, but only four are directly relevant to 
the NRS: obtaining employment, retaining employment, achieving a GED or high school 
credential and entering postsecondary education. The NRS allows programs to designate 
additional goals for students as a main or secondary goal. Note that the default goal of 
educational gain remains regardless of whether the student designates any of the additional goals. 

When a student has one of the “follow-up” goals, the program is held accountable for 
helping the student attain the goal, according to NRS policy. The program or state must obtain 
information on whether the student achieved the goal after he or she leaves the program (see the 
next section on follow-up measures). For this reason, it is important not only that the student can 
attain the goal during the program year, but that the program’s instruction and services are 
oriented toward helping the student achieve the goal. For example, a student with a goal of GED 
attainment should be at a literacy level that will make passing the GED tests likely within the 
year. The student should also receive instruction that will help him or her acquire the additional 
skills needed for passage of the tests. Similarly, if the student’s goal is to obtain a job, the 
program should provide instruction and services to help the student obtain the skills needed to 
obtain employment. 

While setting a realistic goal is important for the NRS, students’ longer-term goals should 
not be ignored simply because they are not obtainable during the NRS reporting period. Many 
programs set short- and long-term goals for students, so that the programs’ performance 
measures will not be adversely affected while the students focus on their longer-term goals. 

Procedures and Policies 

Your program’s goal setting should include the following procedures. Use the 
assessment evaluation form and checklist to rate your program in each area. 

1. Procedure to orient students and help them set goals for instruction. 

Your program should have a procedure in place for orienting students to the program 
and helping them set goals for instruction. The best time for this process to occur is 
when the learner first enters the program. During the intake process, students 
typically complete paperwork, providing basic information about themselves, and 
may take mandated assessments. Either as part of the intake process, or at a separate 
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time shortly thereafter, your program should have another process whereby students 
meet with teachers or an intake counselor at least once to help them establish goals 
for instruction. 

2. Procedure includes identification of attainable short - and long-term goals. 

During the goal setting process, learners, with assistance from program staff, should 
identify their goals and reasons for attending the program. As part of this process, 
staff should help learners set both a realistic timeline for attaining each goal and a 
means for determining whether the goal has been achieved. Setting the timeline and 
evidence of achievement will help the learner realize whether the goal is short- or 
long-term and whether the goal is achievable. For example, when they enter a 
program many learners state very broad goals, such as attaining a GED or getting a 
job. Breaking the goal down into discrete steps — with short and long-term milestones 
along the way — will establish a series of goals which will help learners and teachers 
to design instruction, as well as identify the appropriate goals for NRS purposes. 

Since learners often change their goals as they begin instruction, you might consider 
extending goal setting over additional sessions during the first few weeks. 

3. Method identifies NRS follow-up goals appropriately. 

The goal setting process will identify whether you should set one or more of the NRS 
follow-up goals — obtain employment, retain employment, enter postsecondary 
education or obtain a GED — for a learner. Since your program will be held 
accountable for whether the learner achieves this goal, you should consider, through 
your goal-setting process, whether there is a realistic chance that the student can 
achieve the goal during the reporting period. For example, it is unlikely that lower 
level students will enter postsecondary education or pass the GED tests during this 
short period of time. For such students, you should consider setting such goals for the 
long-term. Employment goals may be more difficult to evaluate. During the goal 
setting process, explore with the learner the skills he or she needs to get a job and the 
time that might be needed to attain these skills. These skills may be broad, such as 
learning to speak English better, or more specific, such as to read manuals to operate 
machinery. 

On the other hand, it may be tempting to be too conservative in setting long-term 
goals, as they are rarely met by learners. You should avoid this inclination, since this 
does a disservice to the learner, and set goals appropriately. Good instruction 
requires you to assist the learner in achieving goals, not play a numbers game. In 
addition, if you ignore long-term goals, you deny your program the opportunity to 
demonstrate that it can help learners achieve such goals. 

4. Procedures for supporting student follow-up to determine goal attainment. 

Once a student has selected an NRS follow-up goal, your program or state is 
responsible for determining whether the goal is attained for NRS reporting. This 
procedure involves following up on the student at some time, depending on the goal, 
after the student has left the program. Your state or program may collect goal 
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attainment through a data matching methodology or by a survey. If your state uses 
data matching, you may need to collect the student’s Social Security number and 
obtain consent to use the number for matching purposes. If your program uses a 
follow-up survey, it is vitally important that you obtain the information you need 
from the student to allow you to locate and contact the student in the future. If you 
are unable to locate the student after he or she has left to obtain information on goal 
attainment, the validity of your data may suffer. Consequently, your intake process 
should include an explanation of the survey and its purpose, contact information and 
the names and numbers of people who might be able to locate the student several 
months later. If you have knowledge of when the student will leave or complete 
instruction, it is also advisable that you remind the student about the survey and re- 
check contact information, shortly before the student exits. 

5. Forms and procedures to enter student demographic and goal information into 
your program’s database. 

To enable your program to check on students with follow-up goals, your program’s 
database must have the capability to retrieve the identity of these students and their 
contact information. Your program’s intake forms and database should have fields 
that allow for the identification of follow-up goals and, if a survey is planned, the 
names, addresses and phone numbers of students and other information on how to 
reach them. 

6. Procedures to obtain demographic and other student measures. 

Programs collect other NRS -related information about learners at intake. Besides 
goals and Social Security numbers (in some programs), programs collect the NRS 
demographic measures (age, race, sex) and employment and disability status. Your 
program should have forms that include all of the NRS and additional state measures 
and the appropriate categories for each measure (such as ethnicity categories) and 
fields in your program database to allow for data entry and retrieval. In addition, 
your program should have a means for timely identification of missing information, 
so that the student can provide the missing data before he or she leaves. 

Indicators of Performance 

If you uncover problems with your program’s intake and goal setting process or simply 
want to examine your program’s procedures further, there are several ways to investigate. You 
should maintain ongoing communication with staff that is involved in goal setting, observe staff 
or talk to students about their goals. You can also review your program’s data for clues on what 
is not working. 

Discuss with staff. Talk to intake staff members about how they assist students in setting 
their goals. You might have staff members discuss or complete the goal setting evaluation form 
to facilitate discussion. If >our discussion uncovers issues or problems, work with staff to 
identify a plan for improving areas of weakness. 
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Observe goal setting. Observe staff during an intake and goal setting session, using the 
evaluation form during your observations. Identify areas of strength and weakness and discuss 
with staff. 

Talk with students. Talk to students individually or in a small group about their 
experience in goal setting, why they set the goals they did and get their evaluation of the goal 
setting process. This discussion may give you insights in the process from the students’ 
perspectives. 

Review program data. Review your program’s student demographic and goal data to 
evaluate whether they provide a clear and accurate picture of your students’ goals, consistent 
with your program’s approach and students’ needs. For example, you could examine: 

❖ The number and percentage of students with specific goals. 

❖ Goals by type of student groups. 

❖ Goals by educational functioning level. 

A review of these data may reveal patterns that stand out or that are difficult to interpret. 
For example, do students at beginning levels have GED attainment as a goal? Do some students 
have no goals listed? Is the number of students with certain goals too low or too high? Also 
examine whether there is an appropriate number of students with a goal of obtain employment 
and enter postsecondary education. 



Intake and Goal Setting: Vignettes 

❖ A community college literacy program in rural Ohio ran classes over a large 
geographic area and wanted to improve the accuracy of its data and insure the 
integrity of its intake process, goal setting and testing. The college administrators 
found staff turnover and, in some cases, indifference to data collection, required 
them to be continuously training and tracking down student files. To ensure 
accuracy of data, the college created an intake and testing team that conducted the 
entire intake, goal setting and testing process at all locations — at the college and 
off site. The completeness and validity of goal and outcome data improved 
markedly. 

❖ The Framingham, Massachusetts Adult ESL Program offers classes to help 
students improve their reading and writing skills and to prepare them to take the 
GED tests. Students set goals at intake and teachers conduct an end of the 
semester interview with students to determine if goals have been met. Goals 
sheets have been translated into various languages so that students completely 
understand what goals they are establishing and what they are being asked to 
report at the end of the semester. 
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Sample Data Table #5 

Intake and Goal Setting: Number of Students Setting Goals by 

Student Group 





Entering Employment 


Retained 

Employment 


Placement in Post- 
Secondary Education 


Number 
with Goal 


Percent 
With Goal 


Number 

with 

Goal 


Percent 
with Goal 


Number 
with Goal 


Percent 
with Goal 


American Indian or Alaskan 
Native 


12 


24% 


15 


30% 


5 


10% 


Asian 


16 


25% 


14 


21% 


57 


89% 


Native Hawaiian or Other Pacific 
Islander 


2 


100% 










Black or African American 


17 


20% 


15 


18% 


3 


4% 


Hispanic or Latino 


89 


12% 


67 


9% 


0 


0 


White 


25 


20% 


21 


17% 


40 


32% 


Total 


161 


15% 


132 


12% 


105 


10% 



Potential Problems 

• Does it make sense that the percentage of Asian students who report having the goal 
of entering post- secondary education is so high? This finding should be checked. 

• It seems unlikely that no Hispanic or Latino students and no African American 
students had the goal of entering post- secondary education. These data should be 
checked. These missing data may represent an omission of collected data or a 
problem collecting these data from this population. 

• The percentage of Hispanic students setting goals is lower than the other ethnic 
groups and should be investigated. 

• 100% of Native Hawaiian or Other Pacific Islanders setting an employment goal 
should be interpreted cautiously, as this group consists of only two students. 
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Sample Data Table #6 

Intake and Goal Setting: Number of Students with Goal by ABE Level 





Total 

Students 

Setting 

Followup 

Goals 


Percent 

with 

Enter 

Employ- 

ment 

Goal 


Number 

with 

Enter 

Employ- 

ment 

Goal 


Percent 

with 

Retain 

Employ- 

ment 

Goal 


Number 
with 
Retain 
Employ- 
ment Goal 


Percent 
with Enter 
Post- 

Secondary 

Education 

Goal 


Number 
with Enter 
Post- 

Secondary 

Education 

Goal 


Beginning Literacy 


427 


19% 


81 


10% 


41 


4% 


17 


Beginning Basic 


441 


5% 


20 


3% 


12 


1% 


3 


Low Intermediate 


529 


25% 


130 


18% 


96 


1% 


5 


High Intermediate 


67 


19% 


13 


10% 


7 


— 


— 


Low ASE 


498 


18% 


91 


7% 


37 


1% 


4 


High ASE 


790 


20% 


157 


11% 


83 


1% 


8 


Total Students with Any 
Follow-up Goals 


2752 


18% 


492 


28% 


768 


5% . 


125 



Potential Problems 



• Are the reported goals set realistic and correct for each level? Seventeen students in 
Beginning Literacy have the goal of entering post secondary education. Is this a 
realistic goal? Not likely; these data should be checked. 

• Is it realistic that such small percentages of students should have a goal of enter 
postsecondary education? These numbers should be checked. 

• The totals should represent the sum of the columns, 125 is not the sum (nor 5% the 
percentage) for the post- secondary education column. 
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Intake and Goal Setting Procedure Evaluation 



A. Describe Your Intake and Goal Setting Procedures 

Describe your program’s student intake process, focusing on how and when student’s 
goals are set, especially NRS outcome goals. Also describe how your program collects 
demographic data (age, race, sex ethnicity). Then complete the checklist in Part B. 

B. Rate Your Intake and Goal Setting Procedures 



Intake and Goal Setting Checklist 


Have 

in 

place 


Have, 
but need 
resources 


Have, 

needs 

change 


Do 

not 

have 


1 . 


Procedure to orient students and help them set goals for 
instruction. 










2. 


Procedure includes identification of attainable short- and long- 
term goals. 










3. 


Method identifies NRS follow-up goals appropriately. 










4. 


Procedures for supporting student follow-up to determine goal 
attainment. 










5. 


Forms and procedures to enter student demographic and goal 
information into your program’s database. 










6. 


Procedures to obtain demographic and other student measures. 











C. Evaluate Your Intake and Goal Setting Procedures 

Identify the aspects of your program’s student intake and goal setting procedures your staff 
misunderstands or lacks knowledge about. Describe your program’s areas of strength and 
weakness, by reviewing the areas you rated as “needs change” and “do not have.” If you checked 
any areas as “need resources,” determine what resources you are lacking and how you might get 
them. Also, review the accuracy and completeness of your description of procedures in Part A on 
the form. After discussing issues with staff, develop a plan for improving weaknesses. 

Areas of Misunderstanding or Lack of Knowledge 
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Areas of Strength 



Areas of Weakness 



Plan for Improvement 



Problem 


Plan for 

Correcting Problem 


People 
Responsible 
for Correcting 


Timeline for 
Completing 
Corrections 


1 . 








2. 








3. 
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Follow-up Measures: Employment, 

GED and Post-Secondary Education 

The NRS includes four outcome measures that must be collected from students after they 
leave the program: entered employment, retained employment, entered postsecondary education 
and obtained a secondary credential (adult high school diploma or pass the GED tests). Programs 
must collect these measures only from students who designate these outcomes as one of their goals 
for attending the program (see the previous section on goal setting). NRS guidelines give states the 
option of collecting these measures through a data matching procedure or through a survey of the 
students who designated the follow-up goals. The NRS allows either the state or local program to 
conduct the survey, as long as the survey methodology includes all local programs. This section 
discusses the NRS procedures for conducting the follow-up survey and for data matching. 

Most programs and states consider conducting the follow-up survey the most difficult 
aspect of NRS data collection. It is extremely difficult to conduct a survey in a way that 
produces valid and reliable results. The process includes determining which students you must 
include in the survey; sampling students, if necessary; locating them and securing their 
cooperation; and administering the survey. Finding the students and getting them to cooperate in 
the survey is critical to its success, since the response rate — the proportion of students you 
reach — largely determines the validity of the information. Locating adult education students is 
especially difficult, given the transient nature of many adult education students. 

Further compounding the difficulties of conducting a survey for NRS purposes are the 
time requirements for the employment measures. The NRS requires that you contact students 
with employment goals during the first quarter after they exit and, if they obtain employment, 
again two quarters later. 

Due to the inherent difficulties of conducting a follow-up survey, many states use data 
matching to collect follow-up measures, especially the employment measures. Data matching 
links records from the program database to another database that has the needed information on 
the same people. For example, using Social Security numbers, student records from a program 
can be matched to the state unemployment insurance wage database to determine whether 
students are employed and have retained employment. Data matching, which is done at the state 
level, removes the burden of the survey from local programs. However, there remain several 
data collection issues that local programs must address to ensure quality data with data matching. 
This section first presents issues related to conducting the follow-up survey and then concludes 
with data matching. 

Procedures and Policies for the Follow-up Survey 

Your program’s procedures for conducting follow-up surveys should include the following 
steps. Use the assessment evaluation form and checklist to rate your program in each area. 
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1. Method for identifying students who must be followed-up. 

According to NRS Guidelines, programs must follow-up on students who designate as a 
main or secondary goal to obtain or retain a job, enter postsecondary education or obtain a 
secondary credential. This follow-up takes place after the student leaves the program. 
Consequently, your program’s database must have the ability to identify students who 
must be followed, which include: (1) all students with a goal of obtaining a job who 
exited, (2) students with a goal of keeping or improving their current job who exited, (3) 
students with a goal of obtaining a secondary diploma or passing the GED who exited and 
(4) students with a goal of entering postsecondary education or training who exited. The 
report or output produced by your program’s database should include student identifying 
and contact information, the student’s follow-up goal and for employment measures, the 
date the student left the program. You should retrieve this information quarterly or 
according to the time you administer the survey. 

2. Sampling procedures in place, if necessary. 

If you have 300 or fewer students in any follow-up outcome group who exited, NRS 
Guidelines require that you include all of these students in the survey. However, if 
you have more than 300 students in any group, you may draw a simple random 
sample of these students. You may use any simple randomization procedure to draw 
the sample, such as drawing every third or fourth name from a student list, a table of 
random numbers, or you can have your program’s computer system draw a random 
sample. If your program has from 301 to 5,000 students in any outcome area who 
exited, the minimum sample size you must draw is 300 for that group. If your 
program has over 5,000 students in any outcome area that exited, you should draw a 
minimum sample size of 1 ,000 for that group. 

3. Survey is conducted at the proper time. 

While you may collect and report attainment of a GED or secondary credential and 
entry into postsecondary education at any time after the student exits, the employment 
measures are tied to specific quarterly exit periods. Students with a goal of obtaining 
a job must obtain the job within the first quarter after program exit quarter. You must 
then collect retained employment on those students who obtained a job two quarters 
later, that is, in the third quarter after program exit. Due to the time specific nature of 
the employment measures, quarterly survey data collection is recommended. 

Note, however, that there are three special cases to these requirements: (1) if a student 
obtains a job while enrolled, you may count that outcome, but only after the student 
exits the program and in the first post-exit quarter; (2) an employed student who 
enters the program with job retention as a goal is surveyed in the third quarter after 
exit quarter to verify continued employment (not in the first exit quarter); and (3) 
retained employme nt is not counted for any student exiting in the third or fourth exit 
quarter of the program year, since their third quarter after exit is beyond the reporting 
period for the program year (i.e., December 31). The following chart summarizes the 
quarterly time periods for collecting employment measures. 
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Exhibit 4-1 

Quarterly Periods for Collecting Entered and Retained Employment 



Exit 

Quarter 


Collect Entered 
Employment by the end of: 


Collect Retained 
Employment by the end of: 


First Quarter 
(July 1-September 30) 


Second Quarter 


Fourth Quarter 


Second Quarter 
(October 1-December 31) 


Third Quarter 


First Quarter, 
Next Program Year 


Third Quarter 
(January 1-March 31) 


Fourth Quarter 


Not Reported 


Fourth Quarter 
(April 1-June 30) 


First Quarter, 
Next Program Year 


Not Reported 



4. Program uses state approved survey instrument, translated for ESOL students, 
if necessary. 

In any survey, how the questions are asked may influence the responses. Therefore it 
is important that the survey questions you ask do not bias or affect responses. For 
comparability of data across programs in the state, it is also highly advisable that all 
programs in the state use the same or equivalent survey instruments. For these 
reasons, you should use a survey questionnaire that your state approves or has 
provided to you. The survey should be short and simple. It is not necessary to have a 
long or complicated survey to collect NRS measures. For example, you need only 
ask if the person got a job or passed the GED. In addition, if you will be surveying 
ESOL students, the survey should be translated into the most common languages your 
students speak. 

5. Adequate staff to conduct survey. 

Conducting a survey is highly labor intensive. Besides administering the survey, you 
must locate the students, explain the survey to them and obtain their cooperation. 

This work requires frequent callbacks to students and careful record keeping. Your 
program should ensure that you have sufficient staff and time to conduct the survey. 
Due to lack of resources, >our program may use teachers or other program staff to 
conduct the survey. However, this approach may be inadequate if these staff 
members do not feel the work is a priority or if they do not have sufficient time to 
conduct the survey. A better approach is to have staff whose primary responsibility is 
to collect the follow-up data. For example, your program could hire a part-time 
assistant or volunteer for the survey. Another approach is to contract the survey out 
to a third party, which several states currently do. While costly, this approach is 
desirable if your program or state can afford it, since it removes much of the burden 
from your local programs. 

6. Staff trained on survey procedures. 

While other methods are possible, most programs will conduct a telephone survey of 
students. Like any other data collection effort, staff must follow a uniform set of 
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procedures to collect data in a valid and reliable manner. You must provide training 
to all staff conducting the survey, which should cover the following topics: what to 
say to students to introduce the survey and get their cooperation; ways to avoid 
refusals; how to ask the survey questions; how to record responses; and how to 
answer student questions about the survey. During the training, you should go over 
every question in the survey to ensure staff understands the purpose of the question, 
what is being asked and what responses are desired. The training should also include 
conducting mock interviews and other practice. Staff should be thoroughly familiar 
with all questions and procedures before beginning. It may be desirable to have a 
follow-up training to ensure understanding and standardization of procedures. 

7. Survey procedures in place to improve response rate. 

The validity of a survey depends to a large degree on the response rate — the 
proportion of people who respond to the survey out of the total number whom you 
targeted to reach in the survey. If you try to ask 100 students whether they passed the 
GED, for example, but only reached 10, you cannot be confident that these 10 
students reflect the other 90. The NRS requires a minimum response rate of 50 
percent. For example, if you have 300 students in the survey, you must get 
information from at least 1 50 students. Getting a good response rate is probably the 
most difficult part of conducting a survey, since it is hard to reach people and get 
them to respond. It may be especially hard to reach adult education students since 
many are transient and may not have telephones. Your survey procedures and 
training should include ways for improving response rate. For example, these 
procedures help improve response rate: 

• Inform students when they enroll and again before they leave about the survey. 
You should also maintain current contact information on students by periodically 
verifying the information with them, especially if you have advance notice of 
when they are leaving the program. 

• Call back students you cannot reach at different times during the day (not just on 
weekday evenings, for example). 

• Leave a detailed message if you cannot reach the student, explaining why you are 
calling and asking for a good time to call back. 

• Stress the importance of the survey to the adult education program, if the student 
is reluctant to participate. 

• Keep the survey short (e.g., 5-10 minutes), so the student does not feel burdened. 

• Keep track of the days and times that students have been contacted. 
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8. Program or state has a database and procedures for entering and reporting 
survey data. 

Once you have administered the survey, you must have a way to record responses so 
that they can be entered into your program or state database. Your state may have a 
special database established for the survey or you may be required to enter the survey 
information directly into your student record database. In some programs and states, 
you might send the survey forms to a data processing contractor who maintains the 
data. Regardless of approach, you need to have an organized method to keep track of 
which students are to be contacted, which students have been reached and whether the 
students achieved the outcomes. You need this information to conduct the survey and 
track your response rate. The state needs the information so it can aggregate the data 
across programs for NRS reporting. You should report in the database the total 
number of students with the goal who exited; if you sampled, the number of students 
sampled; the total number of students you reached; and the number who attained each 
outcome. You can compute your response rate by dividing the number you reached 
over the total number of students (or number sampled, if you sampled) with the 
outcome goal who exited. Exhibit 4-2 shows a table of the type of information your 
database needs to report. 



Exhibit 4-2 

Sample Local Program Survey Report Form 



Outcome Measure 


Total Students 
with Main or 
Secondary Goal 
Who Exited 


Total 

Number of 
Students 
Sampled 


Total 

Number of 
Students 
Responding 


Number 

Achieving 

Outcome 


Response Rate 
(Total 

Responding/ 
Total Sampled) 


Entered Employment 












Retained Employment 












Receipt of 

Secondary Credential 












Entered Postsecondary 
Education or Training 













Indicators of Performance for the Follow-up Survey 

If you uncover problems with your program’s survey process or want to examine your 
survey procedures further, there are several ways to investigate. You could begin by discussing 
the process with staff that is involved in conducting the survey, observe staff s performance and 
review your program’s data for clues on what is not working. 

Discuss with staff. Talk with survey staff members about the problems they have with 
the survey, what works well and what does not. It is helpful to identify staff members who do a 
good job reaching students and getting them to participate. These staff members can help others 
who are not as successful and will give you clues on how best to find students. For example, you 
might discuss what times to reach students, what to say to them and what types of messages to 



Chapter IV. Collecting NRS Outcome Measures 



53 



Improving NRS Data Quality 



leave. Review the evaluation form with staff and work with them to identify a plan for 
improving areas of weakness. 

Observe the survey process. Select times to review with staff the steps they perform 
when conducting the survey, from obtaining the list of students to contact; drawing the sample, if 
applicable; efforts to reach students; and the actual survey administration. Observing the process 
may give you insight into how well the process works and what needs improvement. 

Review program data. Review your program’s student data on follow-up measures to 
evaluate whether it provides a clear and accurate picture of your students’ follow-up outcomes. 
For example, you could examine: 

❖ The number and percentage of students who attain each outcome, overall, by student 
demographics and student educational functioning levels. 

❖ The response rates to the survey overall, by student characteristics and by outcome. 

A review of data like these may reveal patterns that stand out and raise questions. For 
example, are there very low response rates for some types of students? Are there few successful 
outcomes for some measures? You might also look to determine whether the number of students 
achieving each outcome is where you want it to be, given the goals and purposes of your 
program. For example, you might look at the number of students passing the GED or entering 
into postsecondary education, if helping your students achieve these outcomes are goals of your 
program. 

Procedures and Policies for Data Matching 

Data matching is a technical process that requires your data system to produce specific 
data in a required format. To conduct this process, each student to be matched needs a valid 
Social Security number. Use the assessment evaluation form and checklist to rate your program 
in these areas. 

1. Procedure for collecting and validating Social Security numbers. 

Data matching works by pairing records from different databases for the same student 
using a common identifier — a Social Security number. Consequently, you must 
obtain a valid Social Security number for all students whose data will be in the data 
matching pool. You will usually collect this number at intake and, in some states and 
localities, will need to inform the student about the use of the Social Security number 
for this purpose. Some states may require written permission. It is critical to obtain 
Social Security numbers, since without it, data cannot be matched and no outcomes 
can be reported. Similarly, you need a process to verify the validity of Social 
Security numbers before submitting data for matching. Your program database must 
be able to produce a report to identify students with missing, erroneous or duplicate 
Social Security numbers and you should run this report as soon as possible after 
students enroll. If you wait too long to identify problem numbers, the student may 
have left the program and you may not be able to correct the information. 
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2. Data are in the proper format for matching to the external database. 

While there are several ways to perform data matching, all techniques rely on 
software to link multiple databases and produce the number of matches for each 
outcome area. To perform these operations, the software will require your data to be 
in a specific format that will include the location, size and name of each variable, as 
well as the technical format in which your program database is to write the data. 
Ensure that your program database can produce the data according to your state’s 
specifications and that you submit your data in this format. 

3. Data are for the matching time period. 

Your state will have a time period for data submission, such as quarterly or annually. 
When you create the data for submission for matching, your database should produce 
the records for students who have exited your program during this time period. 

Check your data prior to submission to ensure you do not include students who Eire 
still enrolled or students that exited in other time periods. 

4. Data system can produce individual records with Social Security number, goal 
and exit quarter. 

Successful data matching requires individual student records with three pieces of 
information: a Social Security number, so that data can be linked across data-bases; 
the student goal, (e.g., obtain employment) or separate files for students with each 
goal on which data will be matched, so that the student can be matched with the 
correct database; and the exit quarter for employment outcomes, since the NRS 
requires entered employment to be measured in the first quarter after exit quarter. 
Retained employment must be measured the third quarter after exit quarter (see 
Exhibit 4- 1). Your database must be capable of producing records with at least this 
information, and in your state’s required format (see point number 2, above). 

Indicators of Performance for Data Matching 

If you uncover problems with your program’s data matching procedures or simply want 
to examine your program’s procedures further, you can investigate by discussing procedures 
with your intake and data processing staff. You can also review your program’s data for clues on 
what is not working. 

Discuss with staff. The procedures followed by intake staff and data processing staff 
have the greatest effect on the quality of your data for data matching. For intake staff, discuss 
the importance of recording NRS goals for students and collecting Social Security numbers and 
try to resolve any problems with collecting and verifying numbers from students. Talking to data 
entry and processing staff may give you insight on any difficulties in produc ing data sets, 
identifying exit dates or formatting data appropriately. 
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Review program data. Review the data your program plans to submit for data matching 
before you send them to the state or matching agency. Look for patterns of missing data and 
anomalies, by examining tables such as: 

❖ Frequency tables of Social Security numbers. 

❖ Students entry and exit dates. 

❖ Data formats and layouts. 

❖ Student goals. 

A review of these types of tables will reveal problems such as missing or duplicate Social 
Security numbers, inclusion of students who are still enrolled or who exited in a quarter outside 
of the range date you are submitting, and inclusion of students with other goals. 



Sample Data Table #7 

Follow-up Survey Measures: Follow-up Outcome 
Attainment and Response Rates 



Outcome Measure 


Total 
Exiting 
Students 
with Goal 


Total 

Number of 
Students 
Sampled 


Total 

Number of 
Students 
Responding 


Number 

Achieving 

Outcome 


Percent 

Achieving 

Outcome 


Response 

Rate 


Entered Employment 


185 


0 


185 


169 


91% 


100% 


Retained Employment 


615 


312 


689 


516 


75% 


112% 


Receipt of Secondary 
Credential 


506 


300 


19 


11 


58% 


6% 


Entered Post-secondary 
Education 


165 


0 


132 


78 


74% 


80% 


Total 


1471 


612 


1025 


875 


85% 


70% 



Potential Problems 



• Is it realistic that 91% students met their goal of entering employment or that a 100% 
response rate was achieved? These percentages are extremely high and should be 
checked. 

• Response rates should not equal more than 100% (retained employment). The number 
and percentage of students achieving this goal is also likely in error and should also 
be checked. This erroneous percentage will also make the totals incorrect. 

• For receipt of secondary credential, 6% is an extremely low response rate. This rate 
should be checked for data entry error. If accurate, data collection procedures may 
need clarification or revision. These numbers are too low for accurate reporting. 
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Case Study #4: Follow-Up Survey 

In Alabama, the State Department of Education contracts with Auburn University to 
conduct a telephone follow-up survey for all local adult education programs in the state. The 
survey collects entered employment, retained employment and entry into postsecondary 
education Each local program inputs its student data into the state management information 
system and exports its data to the state server quarterly. The state merges all data and the system 
generates follow-up reports by program, core goal and exit date. 

Alabama developed a simple survey, using the NRS Guidelines as a guide to develop the 
survey questions. The survey is in an ACCESS database that interviewers use to enter student 
responses. Survey staff calls each student three times and at different times of the day to 
improve response rate. Auburn sends the follow-up data back to each local program via e-mail 
and local program staff imports the data back into their local databases. 

Auburn staff recommends the following procedures to minimize survey errors when 
developing follow-up procedures: 

• Train interviewers on the survey forms; 

• Understand survey questions, definitions, and phrases; 

• Read questions exactly as written and follow instructions; 

• Complete interview within designated time frame; 

• Attend to administrative issues as soon as possible; 

• Know the purpose and structure of the NRS; 

• Design database with NRS guidelines in mind; 

• Report follow-up data to locals for input into the database at the local level only; 

• Set deadlines for data to be exchanged; and 

• Learn patience! 

Contact: Jo Smith 
Auburn University 
http ://www. auburn, edu/ 
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Follow-up Survey Procedures Evaluation 

A. Describe Your Procedures for Conducting the Follow-up Survey 

Describe your program’s survey procedures for collecting follow-up measures. Include 
the procedures you use for sampling, if applicable. Also describe the survey instrument you use, 
training given to staff on the survey and ways survey data are entered into your computer system. 
Then complete the checklist in Part B. 

B. Rate Your Follow-up Survey Procedures 



Followup Survey Procedures Checklist 


Have 
in place 


Have, 
but need 
resources 


Have, 

needs 

change 


Do 

not 

have 


1 . 


Method for identifying students who must be followed-up. 










2. 


Sampling procedures in place, if necessary. 










3. 


Survey is conducted at the proper time. 










4. 


Program uses state approved survey instrument, translated 
for ESOL students, if necessary. 










5. 


Adequate staff to conduct survey. 










6. 


Staff trained on survey procedures. 










7. 


Survey procedures in place to improve response rate. 










8. 


Program or state has database and procedures for entering 
and reporting survey data. 











C. Evaluate Your Follow-up Survey Procedures 

Identify the aspects of your program’s follow-up survey procedure that you are not 
performing adequately. Focus on procedures your staff misunderstands or lacks knowledge 
about. Describe your program’s areas of strength and weakness, by reviewing the areas you rated 
as “needs change” and “do not have.” If you checked any areas as “need resources,” determine 
what resources you are lacking and how you might get them. Also, review the accuracy and 
completeness of your description of procedures in Part A on the form. After discussing issues 
with staff, develop a plan for improving weaknesses. 

Areas of Misunderstanding or Lack of Knowledge 
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Areas of Strength 



Areas of Weakness 



Plan for Improvement 



Problem 


Plan for 

Correcting Problem 


People 
Responsible 
for Correcting 


Timeline for 
Completing 
Corrections 


1 . 








2. 








3. 
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Data Matching Procedures Evaluation 

A, Describe Your Data Matching Procedures 

Describe how your program deals with data issues related to data matching, including 
collecting of Social Security numbers, and the capability of your program database to produce 
files the state can use for data matching. Then complete the checklist in Part B. 

B. Rate Your Data Matching Procedures 



Data Matching Procedures Checklist 


Have 

in 

place 


Have, 
but need 
resources 


Have, 

needs 

change 


Do 

not 

have 


1. 


Procedure for collecting and validating Social Security 
numbers. 










2. 


Data are in the proper format for matching to external 
database. 










3. 


Data are for the matching time period. 










4. 


Data system can produce individual records with Social 
Security number, goal and exit quarter. 











C. Evaluate Your Data Matching Procedures 

Identify the aspects of your program’s data matching procedures your stalf 
misunderstands or lacks knowledge about. Describe your program’s areas of strength and 
weakness, by reviewing the areas you rated as “needs change” and “do not have.” If you 
checked any areas as “need resources,” determine what resources you are lacking and how you 
might get them. Also, review the accuracy and completeness of your description of procedures 
in Part A on the form. After discussing issues with staff, develop a plan for improving 
weaknesses. 

Areas of Misunderstanding or Lack of Knowledge 
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Areas of Strength 



Areas of Weakness 



Plan for Improvement 



Problem 


Plan for 

Correcting Problem 


People 
Responsible 
for Correcting 


Timeline for 
Completing 
Corrections 


1 . 








2. 








3. 
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Local Staff Training 

Although you may have all of the procedures for collecting NRS data described in this 
guide, your program still may not end up with high quality data if you lack one key element: a 
system for training your staff. Without training, staff will not know or understand the procedures 
and will implement procedures in incomplete or haphazard ways, hurting data reliability and 
validity. 

It is impossible to over-emphasize the importance of a strong, effective system of 
professional development on data collection issues. All too often, states and programs put much 
effort into developing forms, procedures and database systems and put little thought or resources 
into training. In addition, the training cannot be an afterthought or a one-time session, such as an 
annual workshop that is part of a state conference or a review of the NRS procedures manual. 
While this type of training is necessary, it is not sufficient to produce quality data. The training 
should engage staff and be an ongoing feature of your program — offered on a regular basis at 
convenient times. Training should also be based on the needs of staff, focusing on areas where 
there is lack of knowledge or where performance is poor. Regular needs assessments on data 
collection issues will help identify training topics. 

Procedures and Policies 

Your state or other funding agency may support training on data collection and your 
program may also provide training directly to staff on the specific procedures at your site. Use 
the assessment evaluation form and checklist to rate your program in the following areas, which 
reflect good professional development processes. 

1. Staff has received training on NRS policy and data collection procedures. 

All of your staff should be trained and fully knowledgeable of NRS policy, 
accountability policies specific to your state and locality, and the data collection 
process in your program. Training on data collection should cover not only each 
individual’s job in the process, but include a review of others’ roles and how each 
activity affects the other. The training must be specific and detailed, including such 
mundane topics as completing forms, data entry procedures, error checking, and the 
program’s database system, along with general accountability requirements. For 
example, the training should review intake and goal setting procedures, assessment 
policies and follow-up requirements. You should ensure that new staff members 
receive training soon after they start and that there is opportunity for follow-up and 
ongoing training. 

2. A system of continuous professional development on data collection is in place. 

“One-shot” trainings on any topic generally do not have lasting impact. Staff will 
forget procedures, find they misunderstood some part of the training or may think 
some procedures do not work effectively and not follow them. A continuous system 
of professional development will help resolve these problems and given the general 
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high turnover among adult education staff, will also give you an ongoing mechanism 
to train new staff as they join your program. You should schedule training regularly 
throughout the year and also employ different modalities of training to improve 
effectiveness and impact on data collection procedures. For example, you might 
schedule general workshops, individual peer mentoring, shadowing or project- based 
learning activities. 

3. Training addresses staff needs. 

Although all staff should receive an initial, general training on data collection and 
state accountability and NRS requirements, you should design training according to 
the needs of your staff. Using a periodic formal or informal needs assessment, 
collaborative planning process or review of procedures, such as outlined in this 
Guide, you can identify areas where staff needs or wants further training. Using this 
input to design training will make it more relevant to staff, thereby increasing interest 
and the likelihood that the training will result in improved data collection procedures. 

4. Use effective trainers, who use interactive and hands-on activities, to lead 
training. 

The trainers who provide training to local staff are almost as important as the content 
of the training. Use trainers that your staff respects and recognizes as knowledgeable 
of the data collection process. Otherwise, staff is likely to disregard or ignore the 
training. Select trainers who are articulate, well organized and who respect the 
contributions and input of participants in the training. In addition, the training is 
likely to be more effective if it employs interactive, hand-on activities, rather than a 
lecture or “talking head.” For example, you might ask staff to look at actual data 
tables and try to troubleshoot problems on their own, use peer teaching or try role- 
playing or demonstrations. 

5. Training results in learning and improved practice. 

The goal of professional development is to change staff behavior. For training on 
data collection, the desired outcome is that staff leams and then correctly follows all 
procedures. While it is difficult to determine a cause-effect relationship between 
professional development, learning and behavior change, you can look at general 
patterns in data or observe staff at work to get an indication of training effectiveness. 
For example, after a training on assessment, you could observe staff administering 
tests or review student assessment records. If your database has the capability, you 
could examine the assessment data produced by individual staff members to look for 
clues about procedural differences. Whatever the method you use, you will have a 
better professional development approach if it includes ways to verify staff learning 
and implementation of procedures staff has learned. 

Indicators off Performance 

If you want to examine your program’s training further, there are several ways to 
investigate. You should discuss training with staff that is involved in data collection and observe 
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staff members as they do their job. Your program’s data can also give you clues about what 
training might be needed. 

Discuss with staff. Talk to all staff members involved in data collection about what they 
do, what they are not clear about and what they think they could do better. Through this 
discussion you might also explore areas of resistance among staff and reasons why the process is 
not working as it should. Staff may also have insights into ways to resolve data collection 
problems. You could use this information to work with staff to identify a plan for training to 
improve areas of weakness. 

Observe data collection. Observe staff most directly involved in data collection to 
evaluate whether processes are working as you envision them. For example, observe how data 
are entered, forms completed and missing or erroneous data are resolved. Identify areas of 
strength and weakness and discuss with staff. 

Review program data. Review your program’s data for anomalies, missing data and 
other inconsistencies to get insight into areas where more training may be needed. You should 
examine data tables and look at data entered or collected by individual staff members. For 
example, you could examine: 

❖ Amount of missing data by measure and for individual staff members. 

❖ Dates when data are collected and entered. 

❖ Out-of-range data or data that are impossible — for example, dates in the future, ages 
too old or too young, gender or ethnicity categories that do not exist. 

When you evaluate these data, consider whether your program’s data collection policies and 
procedures are clear and understandable. If procedures are well defined but data are still suspect, 
there may be a need for more and better staff training. 
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Sample Data Table #8 

Local Training: Raw Data Table by Collector (Teacher) 



Teacher and Student 


Pretest 

Score 


Posttest 

Score 


Total 

Attendance 

Hours 


Goals 

Set 


Goals 

Met? 


Smith, P. 












Ab, P. 


32 


51 


45 


Yes 


No 


Carillo, M. 


12 


99 


34 


— 


— 


Dillard, P. 


16 


67 


37 


Yes 


— 


Manj, M. 


21 


73 


41 


Yes 


No 


Rodriguez, C. 












Babek, N. 


— 


12 


39 


No 


Yes 


Cronos, S. 


99 


— 


21 


. 


Yes 


Richka, J. 


— 


— 


36 


Yes 


Yes 


Garcia, J. 












Bono, S. 


45 


53 


19 


Yes 


Yes 


Rios, R. 


49 


61 


21 


No 


No 



Potential Problems 

• Rodriguez is not collecting complete or correct assessment data. All students are 
missing some assessment data, and what has been collected falls out of range. The 
data collection process needs to be explained, clarified or enforced with this data 
collector. 

• The students in Garcia’s class appear to be getting far fewer total attendance hours 
than the students in other classes. This problem may be due to poor record keeping 
and should be explored. 

• Smith’s class is missing goal attainment data. 

• Rodriguez has reported goal attainment for a student with no reported goal and for a 
student with initial goals set missing. 

• Garcia’s students pretest scores seem high compared to the other classes. 
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Case Study #5: Local Staff Training 

The Literacy Assistance Center (LAC) in New York City provides training and technical 
assistance to city literacy providers. LAC offers two workshops to help adult education 
providers with NRS data collection procedures. During the NRS Outcomes Workshop, 
participants define NRS goals and outcomes; learn how to collect outcomes according to NRS 
guidelines; learn the logic used by the state data collection system, ALIES, for outcome 
calculations including educational gain, employment outcomes and other outcomes; and test the 
system logic by working through case scenarios. 

During the Final Report Workshop, participants discuss the way that information is 
calculated in the current year’s final report, define and explain each of the final report fields, 
review specific examples to clarify the logic behind final report fields, review the data check 
reports used to verify and troubleshoot the final report, clarifying the use of each report with 
specific examples, and discuss unresolved issues. 

Examples of the types of scenarios and sample data used in the trainings are illustrated 

below: 

• Outcomes Logic Activity. Participants receive examples of students’ test scores, 
goals and outcomes and work through the NRS logic with each student to determine 
if the combination of goals, outcomes, test scores and dates qualifies the student as 
achieving an NRS outcome. 

• Final Report Logic Activity. Participants review examples of students who have 
achieved outcomes, similar to the outcomes logic activity, but these examples are 
discussed from the perspective of the NRS tables. For example, will this student 
count as completing a level? Will this student’s contact hours count? Will the 
student be counted as achieving the outcome with a goal or without a goal? 

• Data Check Report Activity. ALIES data check reports provide programs with a 
look into why their data appear as it does on the NRS tables. Samples of these reports 
include: students with less than 12 contact hours and students without pretests. The 
activity provides scenarios of problems and then participants determine which data 
check report will help them troubleshoot the problem. 

Contact: Megan Swiderski 
Literacy Assistance Center 
http://www. lacnyc. org 
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Local Staff Training Evaluation 



A. Describe Your Procedures for Local Staff Training 

Describe how your program trains local staff on collecting NRS measures and other data. 
Describe when the training occurs, who attends and who conducts each training. Also include 
your strategies for ongoing training and ways you evaluate the effectiveness of training. Then 
complete the checklist in Part B. 

B. Rate Your Local Staff Training 



Local Staff Training Checklist 


Have 

in 

place 


Have, 
but need 
resources 


Have, 

needs 

change 


Do 

not 

have 


1 . Staff has received training on NRS policy and data collection 
procedures. 










2. A system of continuous professional development on data 
collection is in place. 










3. Training addresses staff needs. 










4. Use effective trainers, who use interactive and hands-on 
activities, to lead training. 










5. Training results in learning and improved practice. 











C. Evaluate Your Local Staff Training 

Identify the aspects of your program’s training that you are not conducting well. 
Describe your areas of strength and weakness, by reviewing the areas you rated as “needs 
change” and “do not have.” If you checked any areas as “need resources,” determine what 
resources you are lacking and how you might get them. Also, review the accuracy and 
completeness of your description of training in Part A on the form. After discussing issues with 
staff, develop a plan for improving weaknesses. 

Areas of Misunderstanding or Lack of Knowledge 
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Areas of Strength 



Areas of Weakness 



Plan for Improvement 



Problem 


Plan for 

Correcting Problem 


People 
Responsible 
for Correcting 


Timeline for 
Completing 
Corrections 


1 . 








2. 








3. 
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Chapter V. The State Role in Data Quality 

The role of the federal government in education programs has traditionally been 
to provide funding, general guidance and regulations on how programs should be 
structured and managed. Many details of implementation, such as instructional content 
and approach, are left to state or local discretion. As a state-administered grant program, 
adult education follows this model, with states receiving a basic grant to provide 
instruction and services in accordance with a set of general requirements. States are 
responsible for establishing how instruction should be provided, types of students to 
serve and other details, within their local jurisdictions . 

The NRS uses this same approach of establishing basic rules, with many of the 
specific details for implementation of these rules left to states. The NRS’s 
Implementation Guidelines and Survey Guidelines define the measures, methods and 
reporting procedures for meeting the accountability requirements of the Workforce 
Investment Act (WIA), but both documents also designate a state role for further 
defining, implementing and even enhancing these requirements. 

This chapter describes the state role in the NRS, reviews state responsibilities and 
as in other chapters, provides a self-evaluation checklist to help assess state procedures. 
The chapter also offers suggestions on ways to provide local program oversight and to 
make changes in your state to improve data quality. 

Areas of State Responsibility in the NRS 

In establishing an accountability system for the adult education program, the 
approach of the NRS is to establish federal guidelines to provide uniform requirements 
across the states, while giving each state the freedom and flexibility to design an 
instructional system that meets its own priorities and the needs of its students. This 
flexibility also permits the state to incorporate its own unique accountability requirements 
required by state agencies and other funding sources. 

The need for uniformity in a national accountability system, coupled with the goal 
of state flexibility, creates an inherent tension within the NRS. If requirements are highly 
uniform, there is little or no room for state prerogatives, but there will be a high level of 
consistency of data across states. In contrast, when requirements have little uniformity, 
states have a great deal of flexibility, but this flexibility allows for little comparability 
across states. The NRS attempts to maintain a balance within this dynamic of 
uniformity^ flexibility by establishing frameworks and general requirements for many 
aspects of the system, but allowing states a level of discretion on how they implement 
some parts of the NRS framework. Along with this discretion, however, states have a 
responsibility for implementing their brand of the NRS in a way that produces valid and 
reliable data, consistent with the requirements and spirit of the NRS. There are four areas 
in which states have discretion — and the responsibility — for implementation: assessment 
policy, the use of optional (non-core) outcome measures, methodology for collecting 
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follow-up measures and database and reporting software. States must also provide 
training, technical assistance and monitoring of local programs on data collection and 
reporting. 

Assessment Policies 

One of the most critical state responsibilities is to set a uniform state policy for 
assessment to measure educational gain. For education gain to be meaningful, local 
programs across the state must administer the same assessments in a consistent, 
standardized way. The NRS framework has six educational levels each for ESL and 
ABE, defined by skill descriptors in reading, writing, numeracy, speaking, listening and 
functional skills. Each level also includes test benchmarks for the most widely used tests 
in adult education. States have the option to use the tests or establish their own 
assessment approach, using the skill descriptors. The assessment approach must meet 
standard psychometric criteria for valid and reliable assessment. Using this framework, it 
is the state’s responsibility to select tests or assessments for pre- and posttesting, establish 
times for pre- and posttesting, train local staff on these requirements and ensure local 
programs follow these procedures. 

Use of Optional (non-core) Measures 

Another example of state flexibility within the NRS is the use of secondary or 
optional outcome measures. While the WIA requires the core outcome measures, states 
have the option of collecting secondary measures to meet their own needs or to 
demonstrate additional measures of program performance. The NRS provides definitions 
and methodologies for collecting these measures to promote validity and reliability and to 
allow for a national aggregation, if enough states report them to the Department of 
Education. States must decide which, if any, of the optional measures to report. 

Follow-up Methodology 

Four NRS core outcomes can only be measured after the student leaves the 
program. Collecting these follow-up measures requires a survey of students after they 
finish or withdraw from adult education or the use of a data matching approach, where 
the outcomes for adult education students reside on another database. The data matching 
methodology links students from the other database with the adult education programs’ 
database, using a common identifier. Unemployment Insurance wage records and the 
state GED test database are the most common sources for data matching for employment 
measures and attainment of a secondary credential, respectively. 

With the survey method, programs or the state must survey students from every 
local program with goals related to the measure (e.g., get a job, advance to community 
college) to determine whether students achieved the outcome. It is the state responsibility 
to select a methodology for each NRS follow-up measure. 
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Database and Reporting Software 

Reporting and use of NRS data requires that every local program have an 
individual student record system in a relational database. In addition, states must have 
software that allows aggregation of data from all local programs into a single state 
database for reporting. However, there is no national NRS student record or reporting 
software. States must select or develop their own software for local and state use that 
produces the NRS data tables for reporting. 

Training, Technical Assistance and Monitoring 

The state has the ultimate responsibility for ensuring that all local programs 
follow the state’s data collection and reporting procedures. To make its system work, 
each state is responsible for ensuring that local programs receive ongoing training on 
NRS policies and procedures, state data collection procedures and reporting 
requirements. States should provide this training at least once annually. It is also 
advisable to have procedures and policies in writing or in electronic media (e.g., a 
website or CD) so that local program staff can reference them when necessary. 

State responsibilities also include providing technical assistance to local programs 
to improve their data collection capabilities and monitoring and oversight of local 
procedures. This monitoring should include regular, ongoing review of local data and 
contact with local staff on data collection issues. 

Evaluating Your State NRS Policies and Procedures 

Your state has now implemented the NRS for several years and probably has 
already implemented many of the required policies and procedures. Some of these 
policies, however, may have been overlooked, implemented poorly or may not be 
working. The evaluation form at the end of the chapter will help you analyze and assess 
your state NRS policies and procedures and identify areas for improvement. 

State NRS Policies and Procedures 

The checklist on the evaluation form lists seven areas where your state should 
have policies and procedures. Each point on the checklist is explained below. 

1. Statewide assessment policy has been established. 



Your state should have an assessment policy that describes the assessments 
local programs must use and the times when programs should pre- and 
posttest students. The assessments may be standardized tests or alternative 
assessment, if the assessment meets accepted psychometric standards for valid 
and reliable assessment, including empirically validated scoring rubrics with 
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high inter-rater reliability. Unvalidated rubrics and checklists, and locally 
developed tests, normally do not meet these criteria and are usually not 
acceptable. The state may have more than one assessment, such as a test for 
ESL students and a different test for ABE students, as long as there are clear 
procedures for when to use each test. The state policy should also designate 
when programs should pretest students and the calendar time or instructional 
hours when programs should posttest students. The policy should also clearly 
state that programs are to use a different form of the same assessment for pre- 
and posttesting. 

2. Follow-up methodology has been established. 

The state must determine a methodology for collecting the NRS follow-up 
measures of entered employment, retained employment, attainment of a 
secondary credential and entry into postsecondary education. States must use 
data matching or a follow-up survey, or both methods to collect these 
measures. For example, a state may use the survey for employment measures 
and data matching to determine which students passed the GED tests. The 
survey must include all local programs, although the state or a third-party may 
conduct it. 

3. Policy on NRS optional measures has been established. 

If the state decides to collect any or all of the NRS optional measures, state 
policy should clearly identify these measures and define them consistently 
with NRS definitions. The policy should also state the methodologies for 
collecting these optional measures, which may include survey, data matching 
or direct report from students while they are enrolled. 

4. State has ongoing training and technical assistance to local programs on 
data collection, reporting and use. 

Since local adult education program staff members collect NRS data, they 
must fully understand policies and procedures if they are to produce quality 
data. It is thus critical to the success of the NRS that states provide training to 
teachers and other local staff involved in collecting and reporting data. This 
training should be ongoing, so that training is available for new staff. Critical 
topics for training include definitions of measures, completing reporting 
forms, conducting assessments and follow-up methods. While training should 
cover the general procedures and methods of the NRS, additional training on 
the importance of data and how to use it is likely to increase data quality. 
When local staff can see how to use data for their own purposes, their data 
collection activities will be more meaningful and they are likely to take more 
care in collecting them. 
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5. Data reporting timelines and formats have been established. 

Your state should have requirements for local programs to report data 
according to a fixed, regular schedule. Programs should submit data to a 
central source, such as the state or district, according to this schedule. State 
reporting periods for local programs should be monthly or quarterly, for if the 
time lag for reporting data is too long, the data will not be reported 
completely, as staff will have a tendency to put off data reporting until the 
deadline. The result will be a high degree of missing and possibly false data. 
Another reason for frequent reporting is that errors or problems will be 
identified and can be corrected on an ongoing basis. If data were reported 
only once or twice a year, it would not be possible to identify errors before it 
was too late to correct them. Similarly, your state should specify the technical 
format in which data are to be submitted that is consistent with state reporting 
software. Otherwise you may waste time trying to work with incompatible 
database systems. 

6. A quality control system is in place to monitor and audit local data 
collection. 

To verify the validity of data and ensure local program compliance with state 
data collection policies, your state should conduct frequent reviews of data, 
immediately after local programs submit it. Monitoring procedures should 
also include regular discussions with local data collection staff, either at state 
meetings or by telephone and e-mail, to discuss problems. To be most 
effective, monitoring should be proactive, non- punitive, and presented as a 
form of technical assistance. With this approach, local staff will be less likely 
to hide problems and cover up mistakes. Monitoring should also include at 
least occasional on-site auditing of data, discussed later in this chapter. 

7. State has software or technical standards for local data collection and 
state reporting. 



To meet NRS reporting requirements, the state must have at a minimum, 
software that is capable of aggregating NRS data from all local programs and 
producing the seven required data tables for federal reporting. To report data 
to the state, local programs must have an individual student record database in 
a relational format. Your state must establish a state database system for local 
programs or provide to programs uniform technical standards for database 
development to allow state reporting. All software should also have the 
ability to produce edit reports and have error checking capabilities to identify 
missing and inconsistent data. 
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Indicators of State Performance 

This review of the state role in the NRS may have made you rethink your state 
policies or may have uncovered omissions or other problems with your current policies. 
To evaluate further your state policies, you may wish to discuss these issues with state 
and local staff. You also should review local program data for clues on what may not be 
working. 

Discuss with state staff. State staff members involved in data collection, NRS 
policy and training are likely to have insight about state policies and implementation and 
how well they are working. Talk to staff about state decisions regarding assessment, 
student follow-up and training to get a better understanding of how policies are perceived 
and implemented. For example, you might explore whether posttesting times are 
appropriate, whether local staff administers tests correctly or how to improve response 
rates to the follow-up survey. Discuss other possible options for improving any areas that 
are problematic. 

Discuss with local staff. Since local staff members implement the policies by 
collecting the data, they can be the best informants about how policies are actually 
translated into procedures. Local staff can tell you what “really” happens when forms are 
completed and students assessed. You may get further insights by observing staff during 
student intake and goal setting or giving tests. Identify areas of strength and weakness 
and discuss with staff. 

Review local program data. One of the best ways to identify problems is to 
review local data. Your review should include examining disaggregated data from all 
local programs separately. If you only look at state aggregated data (i.e., summary data 
from all local programs combined) you will miss important details and clues about what 
the data reflect. Tables you could examine include: 

❖ The number and percentage of students who are pre-and posttested by type of 
student and when they are posttested. 

❖ The percent of students who advance by level. 

❖ The number and percent of students who achieve goals. 

❖ Average attendance hours received by students and number of hours it took 
students to advance and achieve goals. 

Review these tables critically by looking for patterns that raise a question or seem 
improbable, with numbers that seem unrealistically high or low. When examining pre- 
and posttest data, examine the average number of instructional hours students received 
when they were posttested to see whether this number conforms to state policy. Look at 
the percentage of students posttested by type of student to see which students are not 
posttested — and if the percentage posttested seems too high. 
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Similarly, examine data for students who advance a level to look for differences 
by student or program. For example, if it takes an average of 40 hours for students to 
advance a level in one program and 120 hours for students to advance from that same 
level in another program, you will want to look further to explain that difference. 

Looking at goal attainments will give you a hint about goal setting and follow-up 
policies. You might investigate why so few students have a goal of passing the GED in 
one program, compared to similar programs in your state. 

Local Data Auditing 

A more formal way to investigate local program adherence to state policies and to 
study data quality is to conduct a local program data audit. Like a financial audit, a data 
audit involves an onsite review of the actual data forms and files and verification of the 
accuracy and validity of the information on these forms. Often an independent, third 
party conducts the audit, such as an accounting firm or a compliance review agency 
within the state government. Since audits are costly and time consuming, most states are 
unable to conduct them frequently or for many programs. Nonetheless, your state should 
perform at least occasional data auditing of a sample of programs, since this type of 
review is the most accurate way to assess data validity at the local level. Findings from 
the audit will also identify technical assistance and training needs and can help prevent 
future problems. 

The auditing process should include at least four procedures. First, the auditor 
should interview program staff involved in data collection on the procedures they follow, 
particularly on how staff deals with missing and incomplete information, data entry 
procedures and reporting times. The auditor should also review the program’s assessment 
and follow-up procedures to ensure they comply with state policy. 

Second, the auditor should draw a random sample of student records to examine 
for completeness and accuracy. The sample size must be large enough to make 
inferences about the program overall and to accommodate the expected high percentage 
of students whom you will not be able to reach. The auditor should compare the written 
records and information on forms of the selected students with information that is in the 
program’s management information system to ensure correspondence between the 
sources. This review will inform you about whether staff completes forms fully and 
accurately and whether there are problems transferring information on forms to the 
program’s database. 

Next, the auditor should contact the sample of students by telephone to obtain 
verification on key variables such as: 

❖ Attendance — ask students to recall dates of active enrollment and approximate 
frequency of attendance; 
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❖ Tests and assessments — ask students to recall whether they took tests and 
assessments and when they took them; Goals set and reasons for attending 
classes; 

❖ Goals met; and 

❖ Satisfaction with services. 

Your state should prepare a formal protocol and standard script for auditors to 
follow when making these calls to minimize interviewer bias. 

As a fourth step in the auditing process, the auditor should verify attainment of 
follow-up goals with a secondary source, especially if the program uses a survey 
methodology. Compared to data matching, surveys are more likely to elicit socially 
desirable responses. For example, students may inaccurately claim to have obtained a job 
or passed the GED tests because they may believe that attaining these goals is expected 
of them. The auditor should contact a sample of employers to verify that the student is or 
was employed, should review GED data to verify the claims of those students who 
claimed to pass the tests, and check enrollment at community colleges to see whether 
students who claimed to enter postsecondary programs actually enrolled. 

The cost and time of conducting a full-scale audit may make the process 
impractical on a large scale for many states. However, auditing allows the state to 
determine data quality and validity with a higher degree of accuracy than any other 
verification. Consequently, it is an indispensable part of the data collection process and 
should be incorporated in some way into state data quality verification systems. 
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Case Study #6 : Local Program Auditing 

The Massachusetts Department of Education (MDOE) recently began 
implementing local audit procedures to ensure local programs’ data collection is reliable 
and valid. MDOE recently conducted its first audit and plans to conduct others by 
selecting programs randomly or selecting programs that indicate having performance data 
that appears to be too low or too high. The state uses an external auditing firm to 
maintain data confidentiality and independence. 

Once a program is selected, the auditor informs local program personnel of the 
process, what is expected of them, and how confidentiality issues are handled. A month 
is then set for the audit. 

The auditor selects a random sample of student records that is large enough to 
support findings and takes into account the expected large percentage of students who 
will be unable to be reached. Using a standard set of questions developed by MDOE, the 
auditor contacts students by telephone to verify data entered in the state database system, 
including attendance (the students’ recollections of period of active enrollment and the 
frequency of attendance), goals set, goals attained and satisfaction with the services 
received. If a student cannot be reached, a message is left indicating a time for a return 
phone call. If the person cannot be reached via phone, a letter is sent. The auditor has 
two weeks to complete all contacts and interviews with students. The auditing process 
also includes using a data match with the GED database to determine if there are any 
false positives or false negatives among students who had a goal of passing the GED 
tests. 

The auditor provides a preliminary report to MDOE within one week of 
completion of student contacts. MDOE has one week to provide the auditor with any 
questions, comments or concerns about the report. The auditor then submits a final report 
not more than three weeks after student contacts are completed. Local program staff, the 
auditor and MDOE hold a final meeting to discuss the report and any future 
recommendations related to the accuracy, timeliness and quality of the program’s data 
collection and recording processes. 

When conducting audits, MDOE offers the following suggestions: 

• Inform students, preferably at intake, that they may be contacted to confirm 
the nature and quality of the services they receive. 

• To improve response rate, identify any special conditions for contacting 
students who previously articulated limits on being contacted and work out 
protocols for such contacts. 

Contact: Donna L. Cornellier 

Massachusetts Department of Education (MDOE) 

http://www. doe. mass, edu/acls/def ault.htm 
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Sample Data Table #9 

State Policy: ABE Beginning Literacy Student’s Pre- 
and Posttest Scores by Pro gram 



Program 


Number of 
Students 


Number 

taking 

Pretest 


Percent 

taking 

Pretest 


Number 

taking 

Posttest 


Percent 

taking 

Posttest 


Average 

Total 

Attendance 

Hours 


Program 1 


210 


191 


91% 


111 


53% 


62 


Program 2 


141 


180 


128% 


51 


36% 


87 


Program 3 


369 


306 


83% 


133 


36% 


55 


Program 4 


450 


360 


80% 


149 


33% 


65 


Program 5 


272 


272 


100% 


101 


37% 


71 


Program 6 


325 


163 


50% 


20 


6% 


17 



Potential Problems 



• Program 2 reports more students taking the pretest than are enrolled in the 
program. One of these values is incorrect. 

• Program 5 reports 100% of students pretested, which should be checked. 

• Program 6 reports that a very small percentage of students was pre- and 
posttested. This data should be verified, and may either indicate data entry 
errors or that data collection practices in this program need to be improved. 

• The percent of students posttested is extreme for Program 1 and Program 6, 
52% seems high and 6% is low. Both values should be checked. 

• The total average attendance hours for Programs 1 and 6 should be checked. 
The values are low for Program 6 and high for Program 2. 

• The data from Programs 2 and 6 contain questionable data with apparent 
errors. All data from these programs should be checked until the data 
collection practices can be improved. 
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Sample Data Table #10 

State Policy: Further Investigation of Programs 
with High Error Rates from Sample Table #9 





Number 

Students 


Number 

taking 

Pretest 


Percent 

taking 

Pretest 


Number 

taking 

Posttest 


Percent 

taking 

Posttest 


Average Total 
Attendance 
Hours 


Program 2- 

Overall 


141 


180 


128% 


51 


36% 


87 


Teacher 1 


77 


127 


165% 


29 


38% 


164 


Teacher 2 


24 


21 


88% 


8 


33% 


52 


Teacher 3 


40 


32 


80% 


14 


35% 


45 


Program 6- 

Overall 


325 


163 


50% 


20 


6% 


17 


Teacher 1 


125 


64 


51% 


20 


16% 


13 


Teacher 2 


57 


26 


46% 


0 


— 


— 


Teacher 3 


46 


21 


46% 


0 


— 


21 


Teacher 4 


61 


30 


49% 


0 


— 


— 


Teacher 5 


36 


22 


61% 


0 


— 


— 



Potential Problems: 

• The data problems for Program 2 come from Teacher 1 ’s class. The number of 
students taking the pretest (127) is incorrectly reported, and is more than the 
number enrolled (77), as is the average total attendance hours (164). The other 
teachers’ data seem fine. This problem may be due to this teacher not 
understanding the data reporting procedures or simply making data entry 
mistakes. 

• The problems for Program 6 are more widespread and program wide. Four of 
the 5 teachers have missing data and none of the teachers pretested over 61% 
of enrolled students. Only one teacher reported posttest data. The average total 
hours for this program are also very low. This program needs to be evaluated 
and staff need to be retrained in data collection procedures. 
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State NRS Policies and Procedures Evaluation 

A. Describe Your State NRS Policies and Procedures 



Describe your state’s assessment, follow-up and reporting policies, software 
requirements for local and state data analyses, and training and technical assistance 
offered to local programs. Also describe your procedures for monitoring and oversight of 
local data collection and how your state reports are prepared and reviewed. Then 
complete the checklist in Part B. 



B. Rate Your State NRS Policies and Procedures 



State NRS Policies and Procedures Checklist 


Have in 
place 


Have, but 
need 

resources 


Have, 

needs 

change 


Do not 
have 


1. 


Statewide assessment policy has been established. 










2. 


Follow-up methodology has been established. 










3. 


Policy on NRS optional measures has been established. 










4. 


State has ongoing training and technical assistance to local programs 
on data collection, reporting and use. 










5. 


Data reporting timelines and formats have been established. 










6. 


A quality control system is in place to monitor and audit local data 
collection. 










7. 


State has software or technical standards for local data collection and 
state reporting. 











Evaluate Your State NRS Policies and Procedures 

Identify the aspects of your state’s NRS policies and procedures that you are not 
performing well. Describe your areas of strength and weakness, by reviewing the areas 
you rated as “needs change” and “do not have.” If you checked any areas as “need 
resources,” determine what resources you are lacking and how you might get them. Also, 
review the accuracy and completeness of your description of state policies and 
procedures in Part A on the form. After discussing issues with staff, develop a plan for 
improving weaknesses. 
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Areas of Misunderstanding or Lack of Knowledge 



Areas of Strength 



Areas of Weakness 



Plan for Improvement 



Problem 


Plan for 

Correcting Problem 


People 
Responsible 
for Correcting 


Timeline for 
Completing 
Corrections 


1 . 








2. 








3. 
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